2.3 Filesystem Layout, Service Control & Startup/Shutdown

2.3 Filesystem Layout, Service Control & Startup/Shutdown

[!CAUTION] Filesystem Safety Rule: Direct modification, deletion, or movement of files within the $PGDATA directory using standard operating system utilities (rm, mv, cp, echo) will result in catastrophic catalog corruption and permanent data loss. All relational changes, tablespace adjustments, and object deletions must be performed strictly through SQL commands.

To effectively maintain, back up, and troubleshoot a PostgreSQL installation, a database administrator must understand how PostgreSQL maps logical database objects onto physical operating system files, and how to govern server lifecycle states safely.

---

The PGDATA Physical Filesystem Anatomy

When a cluster is initialized, initdb establishes a standardized hierarchy of files and subdirectories within $PGDATA. Each directory performs a dedicated function in maintaining ACID durability, relational storage, and cluster configuration.

$PGDATA/
├── PG_VERSION                 # Plain text file containing the major release (e.g. "16")
├── postmaster.pid             # Lock file recording PID, port, socket path, and shared memory key
├── postgresql.conf            # Primary configuration file
├── postgresql.auto.conf       # Auto-managed parameters (written by ALTER SYSTEM)
├── pg_hba.conf                # Host-Based Authentication security configuration
├── pg_ident.conf              # External user map configuration file
├── base/                      # Subdirectories for individual databases (base/<db_oid>/)
├── global/                    # Cluster-wide shared catalogs (pg_database, pg_authid, etc.)
│   └── pg_control             # The critical cluster control file (checkpoint location, state)
├── pg_wal/                    # Write-Ahead Log segments (16MB files)
├── pg_xact/                   # Commit status log files (2-bit transaction states)
├── pg_tblspc/                 # Symbolic links pointing to external tablespaces
├── pg_stat_tmp/               # Temporary runtime statistics storage
├── pg_subtrans/               # Subtransaction status tracking
├── pg_multixact/              # Shared row-lock status data (MultiXact IDs)
└── pg_logical/                # Logical decoding status and replication snapshots

Deep Breakdown of Key Directories and Files

1. PG_VERSION and postmaster.pid

• PG_VERSION: A single-line text file containing the major PostgreSQL version (e.g., 16). If this version does not match the binary version of the running postgres executable, the server refuses to start.
• postmaster.pid: A critical lock file created by the postmaster upon boot. It records the postmaster OS PID, data directory path, cluster start timestamp, port number, socket directory, and shared memory identifier. Its presence prevents multiple PostgreSQL instances from accidentally starting concurrently against the same physical data directory.

2. base/: Per-Database Object Storage

• Within base/, PostgreSQL creates a subdirectory for every individual database, named after that database's unique Object Identifier (OID) in pg_database (e.g., $PGDATA/base/16384/).
• Inside each database subdirectory, tables and indexes are stored as files named according to their relfilenode (retrievable via pg_class).
• 1GB Segmentation: PostgreSQL splits table data files into 1GB segments by default (e.g., 16390, 16390.1, 16390.2) to maintain compatibility with legacy operating systems and filesystem limits.
• Fork Files:
  • _fsm (Free Space Map): Tracks available free space within each 8KB data page to accelerate INSERT operations.
  • _vm (Visibility Map): Tracks pages where all tuples are visible to all active transactions, accelerating Index-Only Scans and skipping frozen pages during VACUUM.

3. global/ and the Vital pg_control File

• Contains cluster-wide system catalogs that are shared across all databases (such as pg_database, pg_authid for user roles, and pg_tablespace).
• pg_control: Located at $PGDATA/global/pg_control, this binary file holds the authoritative state of the entire cluster. It records the current database state (in production, in shutdown, in crash recovery), the exact LSN (Log Sequence Number) of the latest checkpoint, the current REDO point, transaction ID epoch, and catalog version. You can inspect its contents safely using the command-line utility pg_controldata:

  pg_controldata -D /var/lib/pgsql/16/data
  

4. pg_wal/: Write-Ahead Logging Engine

• Holds the 16MB Write-Ahead Log (WAL) segment files (historically named pg_xlog prior to PostgreSQL 10).
• WAL files use a 24-character hexadecimal naming convention (e.g., 000000010000000000000001), representing the timeline, logical log file number, and segment number. PostgreSQL continuously recycles or archives these segments.

5. pg_xact/: Transaction Commit Status

• Historically named pg_clog prior to PostgreSQL 10.
• Stores the commit status of all transactions across the cluster. Because millions of transactions are executed, status is packed efficiently into a 2-bit bitmap per Transaction ID (XID):
  • 00: In Progress
  • 01: Committed
  • 10: Aborted (Rolled back)
  • 11: Sub-committed (Nested subtransactions)

6. pg_tblspc/: Symbolic Links to External Storage

• When an administrator creates a tablespace on an external SSD or SAN volume via CREATE TABLESPACE fast_storage LOCATION '/mnt/fast_ssd/data';, PostgreSQL creates a symbolic link inside $PGDATA/pg_tblspc/ pointing directly to that external path.

---

Service Management and Control: pg_ctl vs. systemd

Managing the PostgreSQL server daemon involves starting, stopping, reloading, and inspecting cluster state.

1. The pg_ctl Administrative Utility

pg_ctl is PostgreSQL's dedicated CLI front-end for server lifecycle administration:

# Start the cluster with log redirection
pg_ctl -D /var/lib/pgsql/16/data -l /var/log/pgsql/logfile start

# Check the operational status of the cluster
pg_ctl -D /var/lib/pgsql/16/data status

# Send SIGHUP to reload configuration files without restarting
pg_ctl -D /var/lib/pgsql/16/data reload

# Restart the cluster using fast shutdown mode
pg_ctl -D /var/lib/pgsql/16/data -m fast restart

# Promote a standby replica to a primary read-write cluster
pg_ctl -D /var/lib/pgsql/16/data promote

2. Integration with Enterprise systemd

In modern Linux distributions (RHEL, Debian, Ubuntu), PostgreSQL is managed as a systemd service unit (postgresql.service or postgresql-16.service). systemd manages dependencies, automatically restarts failed processes, and integrates with OS-level logging via journalctl:

# Start, stop, restart, or reload the PostgreSQL service
sudo systemctl start postgresql-16
sudo systemctl stop postgresql-16
sudo systemctl reload postgresql-16
sudo systemctl status postgresql-16

# Enable the service to boot automatically on OS reboot
sudo systemctl enable postgresql-16

---

Server Shutdown Modes: Smart, Fast, and Immediate

When shutting down a PostgreSQL instance via pg_ctl stop -m <mode>, the administrator must choose between three distinct shutdown modes, which send different operating system signals to the postmaster:

                              pg_ctl stop -m <mode>
                                        │
         ┌──────────────────────────────┼──────────────────────────────┐
         ▼                              ▼                              ▼
   [ Smart (-m smart) ]          [ Fast (-m fast) ]          [ Immediate (-m immediate) ]
      Signal: SIGTERM               Signal: SIGINT                 Signal: SIGQUIT
  Waits for clients to exit    Terminates client backends    Abrupt kill without checkpoint
  No new connections allowed   Rolls back active queries     Simulates power loss/crash
  Flushes clean checkpoint     Flushes clean checkpoint      Requires WAL crash recovery
  Safe, but can take hours     STANDARD DEFAULT: Safe & fast Crash recovery on restart

1. Smart Shutdown Mode (-m smart / Signal: SIGTERM)

• Behavior: The postmaster disallows new client connections, but waits indefinitely for all currently connected client sessions to disconnect voluntarily.
• Checkpoint: Once all active clients have terminated, the checkpointer flushes all dirty shared buffers to disk and writes a shutdown checkpoint record to pg_control and WAL.
• Tradeoff: Extremely safe for batch jobs, but can prevent server reboots for hours or days if idle client connections (or connection poolers) remain open.

2. Fast Shutdown Mode (-m fast / Signal: SIGINT) — THE DEFAULT

• Behavior: The postmaster rejects new connections and immediately transmits a SIGTERM signal to all active backend processes, aborting their queries and rolling back in-flight transactions.
• Checkpoint: Once backends have terminated, the server performs a clean shutdown checkpoint, flushing all dirty pages and writing a shutdown checkpoint marker.
• Restart Impact: Because a clean checkpoint was completed, the cluster can restart immediately without requiring WAL crash recovery. This is the recommended and default shutdown method.

3. Immediate Shutdown Mode (-m immediate / Signal: SIGQUIT)

• Behavior: The postmaster sends a SIGQUIT signal to all child processes and terminates itself instantly without performing a checkpoint.
• Crash Simulation: This mode directly simulates an abrupt power failure, hardware pull, or OS kernel panic.
• Restart Impact: Because dirty buffers in memory were never flushed to disk, the database is left in a crash-inconsistent state. Upon the next startup, PostgreSQL must perform automatic crash recovery, reading pg_control and replaying WAL records from the last valid checkpoint up to the point of termination before accepting client connections.

---

Exam Tips and Common Pitfalls

• Exam Trap: Default Shutdown Mode: Prior to PostgreSQL 9.5, Smart was the default, but in modern PostgreSQL, Fast mode (-m fast) is the default. If an exam question asks what happens during a standard pg_ctl stop, remember that active sessions are aborted, in-flight transactions roll back, and a clean checkpoint is performed.
• Exam Trap: Immediate Mode Consequences: If an exam scenario describes an administrator executing pg_ctl stop -m immediate, do not choose an answer stating that data is permanently lost. WAL ensures durability! However, the next startup will require WAL crash replay, leading to a delayed startup time.
• Exam Trap: Tablespace Storage: If asked where external tablespace files live physically, remember that only symbolic links reside in $PGDATA/pg_tblspc/; the actual table data files reside on the external filesystem path specified during CREATE TABLESPACE.