6.4 etcd Troubleshooting, Quorum Loss & Database Defragmentation

6.4 etcd Troubleshooting, Quorum Loss & Database Defragmentation

etcd is the authoritative state store for the entire Kubernetes cluster. Every resource definition, namespace, secret, configmap, pod status, and node lease is stored within etcd. Because kube-apiserver is stateless, the health, performance, and durability of the entire cluster depend directly on etcd.

Understanding Raft consensus mechanics, quorum requirements, database maintenance routines (compaction, defragmentation, alarm disarming), and snapshot backup/restoration is a core competency tested heavily on the CKA exam.

---

1. Raft Consensus & Quorum Dynamics

etcd uses the Raft consensus algorithm to ensure consistent data replication across a cluster of nodes. To commit a transaction or elect a new leader, etcd requires agreement from a strict majority (quorum) of active members.

$\text{Quorum} = \left\lfloor \frac{N}{2} \right\rfloor + 1$

Where $N$ is the total number of members in the etcd cluster.

[!IMPORTANT] Odd Number of Members: etcd clusters should always contain an odd number of members (1, 3, or 5). Adding an even member (e.g., moving from 3 to 4 nodes) increases the quorum requirement from 2 to 3 without increasing fault tolerance (both 3-node and 4-node clusters can only survive 1 failure), while introducing additional network overhead.

Quorum Loss Symptoms:

If a 3-node etcd cluster loses 2 nodes, quorum is lost:

• etcdctl reports: Error: context deadline exceeded or rafthttp: failed to find member on leader.
• kube-apiserver returns: HTTP 500 Internal Server Error: etcdserver: no leader.
• Do not describe this as a supported read-only mode. Quorum-dependent etcd operations cannot complete, so Kubernetes API reads or writes may time out or fail depending on the request and cache path.

---

2. Setting Up the etcdctl Environment

In standard kubeadm deployments, etcd is configured with mutual TLS authentication. Running etcdctl without proper credentials will result in connection rejection.

# 1. Export ETCDCTL_API version 3
export ETCDCTL_API=3

# 2. Define standard PKI asset paths
export ETCD_CA="/etc/kubernetes/pki/etcd/ca.crt"
export ETCD_CERT="/etc/kubernetes/pki/etcd/server.crt"
export ETCD_KEY="/etc/kubernetes/pki/etcd/server.key"

# 3. Create a reusable shell alias for the CKA exam
alias etcd-cmd="etcdctl --cacert=$ETCD_CA --cert=$ETCD_CERT --key=$ETCD_KEY --endpoints=https://127.0.0.1:2379"

Essential Health & Cluster Inspection Commands:

# Check health of local or remote endpoints
etcd-cmd endpoint health
# Output: https://127.0.0.1:2379 is healthy: successfully committed proposal: took = 2.15ms

# Inspect detailed status (Database Size, Leader ID, Raft Index)
etcd-cmd endpoint status --write-out=table

# List all members of the etcd cluster
etcd-cmd member list --write-out=table

---

3. Database Maintenance: Space Quota, Compaction & Defragmentation

By default, etcd uses a 2 GiB backend quota. The etcd documentation suggests 8 GiB as the maximum for a normal environment; it is guidance, not a hard parser limit on --quota-backend-bytes. When keys are frequently created, updated, or deleted (e.g., high-frequency node leases and events), etcd retains historical revisions for MVCC (Multi-Version Concurrency Control). Over time, the database file (member/snap/db) grows.

+-----------------------------------------------------------------------------------------+
|                        ETCD DATABASE SPACE EXHAUSTION & RECOVERY                        |
|                                                                                         |
|  1. QUOTA EXCEEDED -> etcd triggers 'NOSPACE' Alarm -> API Server writes rejected       |
|                                    |                                                    |
|                                    v                                                    |
|  2. CHECK ALARMS                                                                        |
|     $ etcd-cmd alarm list                                                               |
|     Output: memberID:1029384756 alarm:NOSPACE                                           |
|                                    |                                                    |
|                                    v                                                    |
|  3. COMPACT HISTORICAL REVISIONS                                                        |
|     # Get current revision number from endpoint status                                  |
|     REV=$(etcd-cmd endpoint status --write-out=json | jq .[0].Status.header.revision)   |
|     $ etcd-cmd compact $REV                                                             |
|                                    |                                                    |
|                                    v                                                    |
|  4. DEFRAGMENT DATABASE (Reclaims free disk space from fragmented pages)               |
|     $ etcd-cmd defrag                                                                   |
|                                    |                                                    |
|                                    v                                                    |
|  5. DISARM ALARM (Restores write operations)                                            |
|     $ etcd-cmd alarm disarm                                                             |
+-----------------------------------------------------------------------------------------+

Step-by-Step Defragmentation Runbook:

# Step 1: Check active alarms
etcd-cmd alarm list

# Step 2: Compact database up to current revision
# Retrieve the current revision:
CURRENT_REV=$(ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  endpoint status -w json | grep -o '"revision":[0-9]*' | cut -d: -f2)

ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  compact $CURRENT_REV

# Step 3: Defragment the database to shrink the physical on-disk file
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  defrag

# Step 4: Disarm the NOSPACE alarm to enable writes
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  alarm disarm

---

4. Disaster Recovery: Current Tool Split

Use etcdctl with endpoint and mTLS flags for live server operations, including snapshot save, endpoint health, compaction, defragmentation, and alarms. Use the offline etcdutl utility for snapshot inspection and restore:

ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /var/lib/etcd-backup.db

etcdutl --write-out=table snapshot status /var/lib/etcd-backup.db
etcdutl snapshot restore /var/lib/etcd-backup.db \
  --data-dir=/var/lib/etcd-restored

Before activating a restore on a stacked kubeadm control plane, move the API server and etcd manifests out of /etc/kubernetes/manifests and wait for their containers to stop. Restore to a new empty host directory. In the backed-up etcd.yaml, change the etcd-data hostPath to /var/lib/etcd-restored, while leaving the container mount path and internal --data-dir=/var/lib/etcd consistent. Restore the etcd manifest, inspect it with crictl, restore the API server manifest, and verify endpoint health plus Kubernetes objects. Multi-member recovery additionally requires unique member names and peer settings.