6.2 Node Failure, Kubelet Diagnostics & Systemd Services

6.2 Node Failure, Kubelet Diagnostics & Systemd Services

Worker nodes provide the compute, memory, storage, and networking capacity for Kubernetes workloads. When a node transitions to the NotReady state or begins evicting pods due to resource pressure, the cluster's capacity degrades and workloads risk downtime.

Because the kubelet runs directly on the Linux operating system as a native systemd daemon rather than a containerized workload, troubleshooting node failures requires fluency in Linux system administration, systemd service management, journal log inspection, and host-level container runtime diagnostics.

---

1. Node Status & Health Condition Lifecycle

The Kubernetes control plane monitors worker node health through the Node Lifecycle Controller inside kube-controller-manager. The node's kubelet regularly renews a Lease object in the kube-node-lease namespace (by default every 10 seconds).

+-----------------------------------------------------------------------------------------+
|                         NODE HEARTBEAT & LEASE RENEWAL TIMELINE                         |
|                                                                                         |
|  Kubelet (Node)  ---> Renew Lease (kube-node-lease/node01) ---> kube-apiserver / etcd   |
|                            [Every 10 Seconds: node-status-update-frequency]             |
|                                                                                         |
|  IF HEARTBEAT FAILS:                                                                    |
|  - T = 0s:   Heartbeat stops (Kubelet crash / host network partition / power failure)   |
|  - T = 40s:  Grace period expires (node-monitor-grace-period)                           |
|              Controller marks Node condition: Ready = Unknown / NotReady                |
|              NodeController applies taints:                                             |
|              - node.kubernetes.io/unreachable:NoSchedule                                |
|              - node.kubernetes.io/unreachable:NoExecute                                 |
|  - T = 300s: default not-ready/unreachable NoExecute toleration expires                                   |
|              Pods on failed node transitioned to 'Terminating'                          |
|              ReplicaSet controller spawns replacement pods on surviving healthy nodes    |
+-----------------------------------------------------------------------------------------+

Node Status Conditions:

Execute kubectl describe node <node-name> and inspect the Conditions block:

---

2. Kubelet Diagnostic & Troubleshooting Runbook

When a node reports NotReady, SSH directly into the affected host to perform targeted systemd diagnostics.

# 1. SSH into the failing worker node
ssh node01
sudo -i

# 2. Check the systemd service status
systemctl status kubelet

# 3. If kubelet is inactive (dead) or activating (auto-restart loop), inspect logs
journalctl -u kubelet -e --no-pager -n 100

Common Kubelet Failure Modes and Solutions:

Failure Mode A: Swap is Enabled on the Host

By default, kubelet refuses to start if swap memory is active on the Linux host (unless kubelet swap support is explicitly configured with failSwapOn: false and an appropriate memorySwap.swapBehavior):

Error: failed to run Kubelet: running with swap on is not supported, please disable swap! or set --fail-swap-on flag to false

Remediation:

# Temporarily disable swap immediately
swapoff -a

# Permanently disable swap in /etc/fstab by commenting out the swap partition
cp /etc/fstab /etc/fstab.pre-kubelet
sed -i '/[[:space:]]swap[[:space:]]/ s/^/#/' /etc/fstab

# Restart kubelet
systemctl restart kubelet

Failure Mode B: Container Runtime Socket Path Misconfigured or Runtime Stopped

If containerd or CRI-O is inactive or crashed, kubelet cannot establish its gRPC connection over the UNIX domain socket:

Error: failed to run Kubelet: validate service connection: validate CRI v1 runtime API for endpoint "unix:///run/containerd/containerd.sock": rpc error: code = Unavailable desc = connection error

Remediation:

# Check container runtime daemon status
systemctl status containerd

# If inactive, restart containerd and verify the socket is present
systemctl daemon-reload
systemctl restart containerd
systemctl enable containerd
ls -la /run/containerd/containerd.sock

# Restart kubelet
systemctl restart kubelet

Failure Mode C: Syntax or Parameter Error in Kubelet Configuration

The kubelet daemon reads its configuration parameters from /var/lib/kubelet/config.yaml. Invalid YAML indentation or deprecated flags will cause immediate startup failure:

# Validate YAML syntax
cat /var/lib/kubelet/config.yaml

# Check systemd drop-in configuration overrides
cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

Failure Mode D: Expired Kubelet Client Certificates

Worker node kubelets authenticate to the API server using client certificates located at /var/lib/kubelet/pki/kubelet-client-current.pem. If automatic certificate rotation fails or certificates expire:

# Check certificate expiration date
openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -dates

# Inspect rotation and pending CSRs before changing credentials
readlink -f /var/lib/kubelet/pki/kubelet-client-current.pem
journalctl -u kubelet -n 100 --no-pager
kubectl get csr

# Do not delete client keys blindly. Back them up, then use the cluster's
# documented kubeadm TLS-bootstrap or node rejoin procedure with a valid token.

Failure Mode E: CNI Network Plugin Directory or Configuration Missing

If /etc/cni/net.d/ is empty or CNI binary plugins are missing from /opt/cni/bin/, the node reports NetworkUnavailable=True and Ready=False with the message cni plugin not initialized.

# Check CNI config directory
ls -la /etc/cni/net.d/

# Check CNI binaries
ls -la /opt/cni/bin/

---

3. Node-Pressure Signals and Eviction Ranking

When a node crosses a configured memory, filesystem, inode, or PID threshold, kubelet first attempts node-level reclamation such as removing dead containers or unused images where applicable. If the signal remains under pressure, kubelet evicts Pods.

For the starved resource, kubelet ranks candidates by:

1. whether the Pod's usage exceeds its request;
2. Pod Priority; and
3. usage relative to the request.

QoS is useful context because BestEffort Pods have zero requests and Guaranteed Pods have requests equal to limits, but kubelet does not simply sort all Pods by the label BestEffort, Burstable, Guaranteed. For example, among Pods over their requests, a lower-priority Burstable Pod can be evicted before a higher-priority BestEffort Pod. Inspect kubectl describe node, Pod events, PriorityClasses, requests, and kubelet logs.

Common Linux hard thresholds include memory.available<100Mi, nodefs.available<10%, imagefs.available<15%, and nodefs.inodesFree<5%; node configuration can override them. Soft thresholds add a grace period.

---

4. Safe Node Lifecycle---

4. Safe Node Lifecycle & Maintenance Procedures

Performing kernel patches, operating system upgrades, or hardware maintenance on a worker node requires a strict cordon-drain-uncordon workflow to prevent workload disruption.

+-----------------------------------------------------------------------------------------+
|                        SAFE NODE MAINTENANCE EXECUTION LIFECYCLE                        |
|                                                                                         |
|  1. CORDON NODE                                                                         |
|     $ kubectl cordon node01                                                             |
|     - Marks node Unschedulable (applies node.kubernetes.io/unschedulable taint)         |
|     - Existing running pods remain untouched; no new pods can be scheduled.            |
|                                    |                                                    |
|                                    v                                                    |
|  2. DRAIN NODE                                                                          |
|     $ kubectl drain node01 --ignore-daemonsets --delete-emptydir-data --force           |
|     - Evicts all standalone and controller-managed pods gracefully.                     |
|     - DaemonSet pods are ignored (they cannot be evicted to other nodes).               |
|     - Warning: Pods using emptyDir will lose local ephemeral data.                     |
|                                    |                                                    |
|                                    v                                                    |
|  3. PERFORM MAINTENANCE (SSH to node01)                                                 |
|     - Upgrade packages, patch kernel, reboot: $ sudo reboot                             |
|     - Verify daemons after boot: $ systemctl status containerd kubelet                  |
|                                    |                                                    |
|                                    v                                                    |
|  4. UNCORDON NODE                                                                       |
|     $ kubectl uncordon node01                                                           |
|     - Removes Unschedulable flag; node returns to active scheduling pool.               |
+-----------------------------------------------------------------------------------------+

[!CAUTION] Critical Drain Flags:

• --ignore-daemonsets: Mandatory if any DaemonSet (e.g., kube-proxy, calico-node, prom-node-exporter) runs on the node; otherwise kubectl drain will immediately abort.
• --delete-emptydir-data: Required if any pod on the node mounts an emptyDir volume; otherwise drain aborts to prevent unacknowledged data loss.
• --force: Bypasses standalone pods that are not managed by a Deployment, StatefulSet, or ReplicaSet (Note: standalone pods will be permanently deleted and not rescheduled).