3.8 Node Taints & Pod Tolerations

Node Taints & Pod Tolerations

While Node Affinity attracts Pods to specific nodes, Taints and Tolerations work in reverse: Taints allow a node to repel a set of Pods, ensuring that arbitrary workloads do not land on specialized, reserved, or unhealthy hardware. Tolerations are applied to Pod manifests, allowing (but not forcing) the Pod to schedule onto nodes with matching taints.

---

1. Taints and Tolerations Mechanics

+-----------------------------------------------------------------------------------------+
|                        TAINTS AND TOLERATIONS MECHANICS                                 |
|                                                                                         |
|   [Node 1: Tainted] (dedicated=gpu:NoSchedule)                                          |
|          ^                                                                              |
|          |                                                                              |
|   +------+---------------------------------------+                                      |
|   |                                              |                                      |
|   [Pod A: No Toleration]                         [Pod B: Toleration for dedicated=gpu]  |
|   --> REPELLED (Cannot schedule on Node 1)       --> ALLOWED (Can schedule on Node 1)   |
|                                                                                         |
|   * Critical Rule: Tolerations ALLOW scheduling; they do not FORCE scheduling.          |
|     To guarantee Pod B ONLY lands on Node 1, combine with Node Affinity!                |
+-----------------------------------------------------------------------------------------+

Applying and Removing Taints via kubectl

# Syntax: kubectl taint nodes <node-name> key=value:effect
kubectl taint nodes worker-gpu-01 dedicated=gpu:NoSchedule

# Applying a taint without a value (key:effect)
kubectl taint nodes worker-02 maintenance:NoExecute

# Inspect taints applied to a node
kubectl describe node worker-gpu-01 | grep -i taints
# Output: Taints: dedicated=gpu:NoSchedule

# Removing a taint (append a minus '-' sign to the effect)
kubectl taint nodes worker-gpu-01 dedicated=gpu:NoSchedule-
kubectl taint nodes worker-02 maintenance:NoExecute-

---

2. The Three Taint Effects Deep Dive

A taint's Effect determines the enforcement behavior applied by kube-scheduler and kubelet:

---

3. Pod Toleration Syntax and Matching Rules

A Pod defines tolerations in spec.tolerations using either the Equal or Exists operator.

Example 1: Exact Match with operator: Equal

Requires exact matching of key, value, and effect:

spec:
  tolerations:
    - key: "dedicated"
      operator: "Equal"
      value: "gpu"
      effect: "NoSchedule"

Example 2: Key Matching with operator: Exists

Matches any taint with the specified key regardless of its value:

spec:
  tolerations:
    - key: "node-role.kubernetes.io/control-plane"
      operator: "Exists"
      effect: "NoSchedule"

Example 3: Universal Match (Match All Taints)

Leaving key empty with operator: Exists matches all keys, values, and effects (used by cluster-critical DaemonSets like CNI and logging agents):

spec:
  tolerations:
    - operator: "Exists"

---

4. tolerationSeconds and Delayed Eviction under NoExecute

When a node is tainted with NoExecute, running Pods that tolerate the taint can specify tolerationSeconds to define a grace period before eviction:

spec:
  tolerations:
    - key: "node.kubernetes.io/unreachable"
      operator: "Exists"
      effect: "NoExecute"
      tolerationSeconds: 300
    - key: "node.kubernetes.io/not-ready"
      operator: "Exists"
      effect: "NoExecute"
      tolerationSeconds: 300

If the node enters NotReady or Unreachable state, kubelet delays eviction for 300 seconds (5 minutes), allowing transient network partitions to recover without triggering unnecessary Pod rescheduling.

---

5. Built-in System Taints Injected by Node Controller

The Kubernetes control plane automatically injects node taints when node conditions degrade:

• node.kubernetes.io/not-ready: Node is unhealthy and not ready to receive Pods.
• node.kubernetes.io/unreachable: Node controller has lost communication with kubelet.
• node.kubernetes.io/memory-pressure: Node memory is exhausted.
• node.kubernetes.io/disk-pressure: Node disk capacity is exhausted.
• node.kubernetes.io/pid-pressure: Node process ID space is exhausted.
• node.kubernetes.io/network-unavailable: Node network CNI is uninitialized.
• node.kubernetes.io/unschedulable: Node is cordoned (kubectl cordon <node>).

---

6. Constructing Dedicated Node Pools: The Bidirectional Isolation Pattern

A frequent CKA exam requirement is dedicating nodes exclusively to a specific workload (e.g. GPU compute for machine learning) such that:

1. Standard workloads CANNOT run on the GPU nodes.
2. GPU workloads CANNOT run on standard CPU nodes.

The Complete 3-Step Implementation

1. Taint the specialized nodes to repel all standard workloads:

   kubectl taint nodes gpu-node-01 sku=gpu:NoSchedule
   

2. Label the specialized nodes for targeting:

   kubectl label nodes gpu-node-01 sku=gpu
   

3. Configure the ML Pods with both a Toleration (to pass the taint) AND Node Affinity / nodeSelector (to force placement on the labeled nodes):

apiVersion: v1
kind: Pod
metadata:
  name: gpu-training-pipeline
spec:
  # 1. Force Pod onto GPU nodes
  nodeSelector:
    sku: gpu
  # 2. Allow Pod to tolerate the GPU taint
  tolerations:
    - key: "sku"
      operator: "Equal"
      value: "gpu"
      effect: "NoSchedule"
  containers:
    - name: model-trainer
      image: tensorflow/tensorflow:latest-gpu