1.7 etcd Snapshot Backup and Restore Procedures

1.7 etcd Snapshot Backup and Restore Procedures

Etcd stores Kubernetes API state, including workloads, RBAC, Secrets, and custom resources. A snapshot is therefore a cluster-state backup, not a backup of application data held in persistent volumes. Practice both saving a snapshot and activating a restore; a backup is useful only when the recovery path is understood.

1. Discover the Running Member Configuration

In a kubeadm stacked control plane, inspect the static Pod manifest rather than assuming paths:

sudo grep -E -- '(advertise-client-urls|trusted-ca-file|cert-file|key-file|data-dir)' \
  /etc/kubernetes/manifests/etcd.yaml

Common local values are endpoint https://127.0.0.1:2379, CA /etc/kubernetes/pki/etcd/ca.crt, and the etcd server certificate and key. The API server's apiserver-etcd-client.crt and key are also client credentials when the manifest shows those paths. Use the files present on the assigned host.

2. Save the Online Snapshot with etcdctl

Snapshot creation contacts the running server, so it requires the endpoint and mTLS credentials:

sudo ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /opt/cluster-snapshot.db

Check the exit status and file before proceeding. For an etcd data directory copied directly from disk, etcd documentation warns that it can omit data still in memory; snapshot save is the normal live-backup method.

3. Inspect the Snapshot with etcdutl

Etcd 3.5 deprecated the etcdctl snapshot status and etcdctl snapshot restore subcommands. The current offline utility is etcdutl:

sudo etcdutl --write-out=table snapshot status /opt/cluster-snapshot.db

Status reports fields such as hash, revision, key count, and size. It does not prove that every external workload volume is recoverable, but it confirms that the utility can read the snapshot metadata. Because this is an offline file operation, endpoint and TLS flags are not part of the status command.

4. Stop Static Pods Before Cutover

Create a backup directory outside the watched static-Pod directory, then move the API server and etcd manifests there:

sudo mkdir -p /opt/k8s-manifests-backup
sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml /opt/k8s-manifests-backup/
sudo mv /etc/kubernetes/manifests/etcd.yaml /opt/k8s-manifests-backup/
sudo crictl ps --name 'etcd|kube-apiserver'

Wait until the kubelet has stopped those containers. Keep backup manifests out of /etc/kubernetes/manifests; kubelet scans files in that directory regardless of a convenient “backup” suffix.

5. Restore into a New Data Directory

The target directory must not already contain an etcd member:

sudo etcdutl snapshot restore /opt/cluster-snapshot.db \
  --data-dir=/var/lib/etcd-restored
sudo ls -la /var/lib/etcd-restored

A restore creates a new logical cluster and member identity. For a multi-member external etcd cluster, each member needs the correct unique name, peer URLs, and initial-cluster settings; do not apply a single-member recipe unchanged. On a kubeadm stacked single member, the manifest already supplies its peer settings.

6. Activate the Restored Directory

Edit the backed-up etcd.yaml. Change the host-side volumes[].hostPath.path for etcd-data to /var/lib/etcd-restored. Keep the container's mount path and --data-dir at /var/lib/etcd unless the manifest deliberately uses a different internal path:

volumeMounts:
- name: etcd-data
  mountPath: /var/lib/etcd
volumes:
- name: etcd-data
  hostPath:
    path: /var/lib/etcd-restored
    type: DirectoryOrCreate

Preserve ownership and permissions appropriate to the etcd image. Move etcd.yaml back first and use crictl ps -a --name etcd plus crictl logs <id> if it fails. Then restore the API server manifest.

7. Validate Recovery

After both static Pods are healthy, test etcd and Kubernetes state:

sudo ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key endpoint health
kubectl get nodes
kubectl get pods -A

8. Recovery Boundaries and Evidence

A restore returns Kubernetes objects to the snapshot revision. Objects created afterward disappear, and later updates roll back. That does not roll back bytes stored on external PersistentVolumes, cloud databases, object stores, or other systems, so applications can require their own consistency procedure. Snapshot files also contain Secret objects and should be protected like sensitive credentials: restrict file permissions, control copies, and use approved encryption and retention policies.

For a multi-member cluster, never start multiple restored members with identical names or peer URLs. Restore the same snapshot separately for each member using a unique --name, matching --initial-advertise-peer-urls, a complete --initial-cluster, and an appropriate token. Bring members up according to the etcd recovery plan and confirm quorum before reconnecting API servers.

Capture evidence at each boundary: snapshot status before recovery; runtime logs when the restored etcd starts; endpoint status --write-out=table and endpoint health; and representative Kubernetes objects after the API server returns. Check namespaces, critical Deployments, RBAC, and any object the task explicitly says must be recovered. A green endpoint alone proves database availability, not application consistency.

Keep the original data directory until the restored cluster has been verified. A restart alone does not select a restored directory; the hostPath cutover is the decisive step.