1.3 etcd Architecture, Clustering & Raft Consensus

1.3 etcd Architecture, Clustering & Raft Consensus

At the foundation of every Kubernetes cluster sits etcd, an open-source, distributed, strongly consistent key-value store created by CoreOS and maintained by the CNCF. In Kubernetes, etcd holds the complete single source of truth: every Namespace, Pod, ConfigMap, Secret, Deployment, CRD, and event object is serialized into Protocol Buffers and persisted into etcd's bbolt database file (member/snap/db).

Because all state mutations pass through etcd, understanding its consensus mechanics, clustering constraints, failure domains, and CLI tooling (etcdctl) is one of the highest-yield subjects on the CKA examination.

---

1. Raft Consensus Mechanics & State Replication

etcd implements the Raft consensus algorithm to ensure that all members in a distributed cluster maintain an identical, ordered log of key-value operations despite node crashes or network partitions.

+-----------------------------------------------------------------------------+
|                        RAFT CONSENSUS STATE MACHINE                         |
|                                                                             |
|   Client Request (Write)                                                    |
|          |                                                                  |
|          v                                                                  |
|   +---------------+                                                         |
|   |  RAFT LEADER  |                                                         |
|   | (Node 1:2379) |                                                         |
|   +---------------+                                                         |
|          |                                                                  |
|          +-------------------+ AppendEntries RPC (Port 2380)                 |
|          |                   |                                              |
|          v                   v                                              |
|   +---------------+   +---------------+                                     |
|   | RAFT FOLLOWER |   | RAFT FOLLOWER |                                     |
|   |    (Node 2)   |   |    (Node 3)   |                                     |
|   +---------------+   +---------------+                                     |
|          |                   |                                              |
|          +-----> [QUORUM: 2 of 3 Acknowledge Write] <-----+                 |
|                               |                                             |
|                               v                                             |
|                  [TRANSACTION COMMITTED TO DISK]                            |
+-----------------------------------------------------------------------------+

Raft Operational Roles:

1. Leader: Accepts all write proposals from client applications (kube-apiserver). The leader assigns monotonically increasing index numbers to entries, writes them to its local Write-Ahead Log (WAL), and replicates entries to all Follower nodes via AppendEntries RPCs.
2. Follower: Completely passive. Forwards write requests to the Leader. Accepts heartbeats and log replication RPCs from the leader. If heartbeats cease within an election timeout window, the follower transitions to Candidate.
3. Candidate: Initiates a new election term, increments term counter, votes for itself, and broadcasts RequestVote RPCs to peer nodes.

Write Lifecycle & Quorum Commit:

1. Kube-apiserver issues a PUT request to the Leader on client port 2379.
2. Leader writes entry to local Write-Ahead Log (WAL).
3. Leader broadcasts AppendEntries RPCs to Followers across peer port 2380.
4. When a Quorum Majority of nodes confirm writing the log entry to their disk, the Leader commits the transaction.
5. The Leader applies the transaction to its key-value state machine (bbolt DB) and returns success to kube-apiserver.
6. Followers apply the committed entry to their local state machines upon receiving the Leader's next heartbeat.

---

2. Quorum Formulation & Fault Tolerance Analysis

Raft guarantees consistency by requiring that any state modification or leader election receive votes from a strict majority of the cluster, known as Quorum.

$\text{Quorum} = \left\lfloor \frac{N}{2} \right\rfloor + 1$

Where $N$ is the total active member count of the etcd cluster. The maximum number of simultaneous failure tolerance ($F$) is calculated as:

$F = \left\lfloor \frac{N - 1}{2} \right\rfloor$

Quorum & Fault Tolerance Matrix:

[!IMPORTANT] The Odd Number Rule: Adding an even member (e.g., going from 3 to 4 nodes) does not increase fault tolerance (both tolerate only 1 failure), but it increases the number of votes required for quorum from 2 to 3. The fourth member adds replication work without increasing tolerated failures. Prefer an odd member count such as 3 or 5 unless a documented transition or topology requirement justifies a temporary even count; quorum loss makes consensus-dependent operations unavailable, not a supported read-only cluster mode.

---

3. etcd Communication Ports & Mutual TLS (mTLS)

etcd isolates administrative client traffic from inter-node synchronization by binding to two distinct TCP ports:

• Port 2379 (Client API): Listens for incoming queries from kube-apiserver and etcdctl.
• Port 2380 (Peer-to-Peer): Listens for internal Raft consensus heartbeats, leader election ballots, and log streaming between etcd cluster members.

Kubeadm configures etcd client and peer communication with mutual TLS (mTLS). Other etcd deployments are configurable, so verify the actual flags and certificates. To execute administrative CLI commands, you must supply the trusted CA cert, client certificate, and private key.

/etc/kubernetes/pki/etcd/
├── ca.crt               # etcd Certificate Authority
├── ca.key               # etcd CA Private Key
├── server.crt           # Server certificate (Port 2379)
├── server.key           # Server private key
├── peer.crt             # Peer-to-peer cert (Port 2380)
├── peer.key             # Peer private key
├── healthcheck-client.crt # Used by the local etcd health probe
└── healthcheck-client.key

---

4. etcdctl v3 Command Operations & Health Diagnostics

On Kubernetes nodes, the etcdctl binary communicates using the v3 API. Ensure ETCDCTL_API=3 is set in your environment.

Essential Diagnostic Commands:

# Export API version:
export ETCDCTL_API=3

# Common TLS flag alias for exam speed:
alias etcd-cmd="etcdctl \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  --endpoints=https://127.0.0.1:2379"

# 1. Check Cluster Member List:
etcd-cmd member list -w table

# 2. Check Endpoint Health:
etcd-cmd endpoint health -w table

# 3. Check Endpoint Status (DB Size, Leader ID, Raft Revision):
etcd-cmd endpoint status -w table

Output Interpretation of endpoint status:

---

5. Compaction, Defragmentation & Space Management

etcd is a Multi-Version Concurrency Control (MVCC) datastore. When a Kubernetes object is updated or deleted, etcd does not overwrite the old data immediately; it appends a new revision so clients can watch historical changes. Over time, old revisions consume disk space.

# 1. Compact old revisions up to revision 35000:
etcd-cmd compact 35000

# 2. Defragment backend storage to release freed pages back to the filesystem:
etcd-cmd defrag --endpoints=https://127.0.0.1:2379

# 3. Disarm space quota alarm if triggered:
etcd-cmd alarm disarm