6.6 Workload Scheduling Failures & Resource Constraint Resolution

6.6 Workload Scheduling Failures & Resource Constraint Resolution

The kube-scheduler is responsible for assigning unscheduled pods (spec.nodeName == "") to the most appropriate worker node in the cluster. When scheduling constraints cannot be satisfied, pods remain stuck in the Pending state indefinitely.

Diagnosing scheduling failures requires understanding the scheduler's internal evaluation pipeline, interpreting predicate failure messages, and resolving conflicting resource requests, affinity rules, taints, tolerations, and storage topology constraints.

---

1. The Two-Phase Scheduling Algorithm

When a pod is submitted to the API server, the scheduler runs two sequential phases to select a target node:

+-----------------------------------------------------------------------------------------+
|                         KUBE-SCHEDULER TWO-PHASE PIPELINE                               |
|                                                                                         |
|  [UNSCHEDULED POD] (spec.nodeName is empty)                                             |
|         |                                                                               |
|         v                                                                               |
|  [PHASE 1: FILTERING (PREDICATES)]                                                      |
|  Evaluates hard requirements. Drops nodes that cannot host the pod:                     |
|  - NodeResourcesFit (Checks CPU, Memory, Inodes, Pod count against Allocatable)         |
|  - NodeName / NodeSelector / NodeAffinity (Label matching)                             |
|  - PodToleratesNodeTaints (Taint / Toleration compatibility)                            |
|  - PodTopologySpread / InterPodAffinity (Anti-affinity rules)                           |
|  - VolumeBinding / VolumeZoneLimits (Storage availability zone matching)                |
|         |                                                                               |
|         +---> [0 NODES SURVIVED] ===> Emit 'FailedScheduling' Event -> Pod remains PENDING
|         |                                                                               |
|         v [1+ NODES SURVIVED]                                                           |
|  [PHASE 2: SCORING (PRIORITIES)]                                                        |
|  Ranks surviving nodes on a 0–100 scale:                                                |
|  - NodeResourcesBalancedAllocation (Spreads resource usage evenly)                      |
|  - ImageLocalityPriority (Prefers nodes with container image already cached)            |
|  - PreferredDuringScheduling Node/Pod Affinities                                        |
|         |                                                                               |
|         v                                                                               |
|  [BINDING] (Highest scoring node selected -> Pod.spec.nodeName committed to etcd)       |
+-----------------------------------------------------------------------------------------+

---

2. Diagnosing FailedScheduling Events

The primary diagnostic tool for scheduling failures is kubectl describe pod <pod-name>. Inspect the Events table at the bottom of the output:

kubectl describe pod pending-app

Common Predicate Failure Patterns & Root Causes:

Pattern 1: Insufficient Compute Resources (NodeResourcesFit)

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  12s   default-scheduler  0/4 nodes available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 3 Insufficient memory.

• Root Cause: The pod's .spec.containers[].resources.requests.memory exceeds the allocatable memory capacity of all eligible worker nodes.
• Key Distinction: Scheduling evaluates requests, NOT limits. Even if a node has 16GB of actual idle RAM, if existing pods have already reserved 15GB in requests on a 16GB node, a new pod requesting 2GB will be rejected.

Pattern 2: NodeSelector & NodeAffinity Mismatches

Warning  FailedScheduling  8s    default-scheduler  0/3 nodes available: 3 node(s) didn't match Pod's node affinity/selector.

• Root Cause: The pod specifies a nodeSelector or requiredDuringSchedulingIgnoredDuringExecution node affinity requiring labels (e.g., disktype: ssd or topology.kubernetes.io/zone: us-east-1a) that are not present on any worker node.

# Check existing labels on all nodes
kubectl get nodes --show-labels

# Add the missing label to a worker node to resolve the scheduling block
kubectl label node node02 disktype=ssd

Pattern 3: Taints & Tolerations Rejections

Warning  FailedScheduling  5s    default-scheduler  0/3 nodes available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 2 node(s) had untolerated taint {dedicated: gpu}.

• Root Cause: Worker nodes have been tainted (e.g., dedicated=gpu:NoSchedule), but the pod manifest lacks the corresponding tolerations block.

---

3. Taints, Tolerations & Node Cordoning Mechanics

Taints allow a node to repel a set of pods unless the pod explicitly tolerates the taint.

+-----------------------------------------------------------------------------------------+
|                              TAINT EFFECT MATRIX                                        |
|                                                                                         |
|  +-------------------+  +------------------------------------------------------------+  |
|  | NoSchedule        |  | New pods without matching toleration CANNOT be scheduled.  |  |
|  |                   |  | Existing pods already running on the node remain untouched.|  |
|  +-------------------+  +------------------------------------------------------------+  |
|  +-------------------+  +------------------------------------------------------------+  |
|  | PreferNoSchedule  |  | Scheduler PREFERS to avoid placing pods without toleration |  |
|  |                   |  | on this node, but will place them if no other nodes exist. |  |
|  +-------------------+  +------------------------------------------------------------+  |
|  +-------------------+  +------------------------------------------------------------+  |
|  | NoExecute         |  | New pods without toleration CANNOT be scheduled.           |  |
|  |                   |  | EXISTING pods without toleration are IMMEDIATELY EVICTED.  |  |
|  +-------------------+  +------------------------------------------------------------+  |
+-----------------------------------------------------------------------------------------+

Taint Management Commands:

# Apply a taint to worker node
kubectl taint nodes node01 dedicated=special:NoSchedule

# Remove a taint (note the trailing minus sign)
kubectl taint nodes node01 dedicated=special:NoSchedule-

# Check taints on all nodes
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

Pod Manifest Toleration Example:

spec:
  tolerations:
  - key: "dedicated"
    operator: "Equal"
    value: "special"
    effect: "NoSchedule"

---

4. Storage Zone Conflicts (VolumeZoneConflict)

In multi-availability zone cloud environments (e.g., AWS EBS, GCP Persistent Disks), cloud block storage volumes are tied to a specific physical availability zone (e.g., us-east-1a).

If a pod binds to a PersistentVolume in us-east-1a, but the scheduler attempts to place the pod on a node in us-east-1b, the scheduler fails with VolumeZoneConflict.

# StorageClass using WaitForFirstConsumer prevents premature volume binding:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: topology-aware-sc
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer  # Delays PV creation until Pod is scheduled

---

5. Manual Pod Scheduling (spec.nodeName)

When kube-scheduler is disabled or failing during an emergency, an administrator can bypass the scheduling pipeline entirely by manually binding a pod directly to a target node.

apiVersion: v1
kind: Pod
metadata:
  name: emergency-debug-pod
spec:
  # Directly assigns pod to node01, bypassing kube-scheduler entirely:
  nodeName: node01
  containers:
  - name: web
    image: nginx:alpine

[!NOTE] When spec.nodeName is set, the scheduler ignores the Pod, so nodeSelector, affinity, scoring, and NoSchedule taints are not evaluated by the scheduler. The named kubelet still tries to admit the Pod and it can fail for unavailable resources; a matching NoExecute taint can eject it unless tolerated.