5.2 Cluster Administration: Node Maintenance, Upgrades & etcd Backup

5.2 Cluster Administration: Node Maintenance, Upgrades & etcd Backup

Quick Answer: Taking a node out of service is a two-step ritual: kubectl cordon stops new Pods landing on it, and kubectl drain evicts what is already there while respecting PodDisruptionBudgets. Cluster upgrades run control plane first, then worker nodes, one minor version at a time. Because kube-apiserver persists every object in etcd, an etcd snapshot is the only complete backup of cluster state.

The official curriculum lists Administration as one of the four competencies under Kubernetes Fundamentals — the largest domain on the exam. KCNA does not ask you to run these commands, but it does ask what they do and in what order.

---

1. Taking a Node Out of Service

# 1. Stop the scheduler placing anything new here
kubectl cordon node-07

# 2. Evict what is already running
kubectl drain node-07 --ignore-daemonsets --delete-emptydir-data

# ... perform kernel patch / hardware swap / kubelet upgrade ...

# 3. Return the node to the pool
kubectl uncordon node-07

Why the Two Flags Are Almost Always Needed

• --ignore-daemonsets — DaemonSet Pods are immediately recreated on the same node by their controller, so drain refuses to proceed unless told to skip them. Almost every real cluster runs CNI, kube-proxy, and log-shipper DaemonSets, so this flag is effectively mandatory.
• --delete-emptydir-data — confirms you accept that data in emptyDir volumes is destroyed. Drain will not silently discard it.
• --force — required for naked Pods (Pods with no owning controller), because nothing will recreate them elsewhere. Reach for it knowingly.

Drain Uses the Eviction API, Not Delete

This is the detail that matters. drain issues Eviction requests, which are checked against PodDisruptionBudgets. If evicting a Pod would breach a budget, the request is rejected and drain retries until a replacement becomes Ready elsewhere. That interlock is exactly what turns a node-by-node cluster upgrade into a non-event instead of an outage — and it is why a PDB with no headroom (minAvailable equal to replicas) blocks drains forever.

---

2. Upgrading a Cluster

Kubernetes enforces a bounded version skew, so upgrades follow a fixed order and a fixed step size.

STEP 1  Control plane node 1: upgrade kubeadm → apply → upgrade kubelet + kubectl
STEP 2  Remaining control plane nodes, one at a time
STEP 3  Worker nodes, one at a time:
           cordon → drain → upgrade kubelet/kube-proxy → uncordon
STEP 4  Verify: kubectl get nodes shows the new version everywhere

The governing rules:

• Never skip a minor version. 1.31 → 1.33 is unsupported; go 1.31 → 1.32 → 1.33.
• kubelet may be up to three minor versions older than kube-apiserver, never newer. A node running a newer kubelet than the API server is an unsupported configuration.
• Upgrade the control plane before the nodes, so the API server is always the most current component.
• Check the deprecation notes first. Removals land at minor releases, so an upgrade is a migration event. Alert on the apiserver_requested_deprecated_apis metric before you begin.

On a managed service the control plane step is a provider action (a console click or API call); the node-pool step remains yours.

---

3. etcd Backup and Restore

Every object — Deployments, Secrets, RBAC bindings, CRD instances — lives in etcd, and kube-apiserver is the only component that writes to it. Therefore:

An etcd snapshot is a complete backup of cluster state. Nothing else is.

# Take a snapshot
ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-2026-08-07.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Inspect it
etcdctl snapshot status /backup/etcd-2026-08-07.db --write-out=table

# Restore into a fresh data directory
etcdctl snapshot restore /backup/etcd-2026-08-07.db \
  --data-dir=/var/lib/etcd-restored

Operational points KCNA cares about:

• A snapshot captures API objects only. It does not capture the contents of PersistentVolumes — application data needs its own backup, typically via CSI volume snapshots or a tool such as Velero.
• Velero is the common ecosystem answer for backing up namespaces plus their PersistentVolumes together, and for cluster migration.
• Store snapshots off the control plane node. A backup that dies with the machine is not a backup.
• Restoring rolls the whole cluster back to the snapshot instant: objects created since then are gone. Restore is a last resort, not a routine.

---

4. Certificates

Kubernetes is mutually TLS-authenticated end to end. kubeadm issues client and serving certificates with a one-year lifetime and renews them automatically on each kubeadm upgrade. Clusters that are not upgraded for over a year hit expired certificates and lock everyone out — a genuinely common outage.

kubeadm certs check-expiration     # what expires when
kubeadm certs renew all            # renew, then restart control plane pods

The cluster CA certificate has a ten-year default lifetime and is not rotated by a normal upgrade.

---

5. Governance Controls for a Shared Cluster

Four mechanisms together make a multi-tenant cluster administrable:

Audit Logging

The API server can emit a structured audit event for each request at one of four levels — None, Metadata (who, what, when, verdict), Request (adds the request body), and RequestResponse (adds the response body). An AuditPolicy maps rules to levels, so you can log Secret access at Metadata while logging routine get pods at None. Audit logs are the primary forensic evidence after an incident and are typically shipped off-cluster immediately.

Node Health

Routine administration also means watching node conditions — Ready, MemoryPressure, DiskPressure, PIDPressure — since the kubelet begins evicting Pods under pressure long before the node goes NotReady. A node that oscillates between Ready and NotReady is usually reporting a kubelet, container-runtime, or network-plugin problem rather than a workload problem.