4.1 Scheduling, Resource Management & Node Selection

4.1 Scheduling, Resource Management & Node Selection

When a Pod is submitted to the Kubernetes API server, the control plane must decide which worker node has sufficient capacity and matching constraints to run that workload. This placement decision is made by the kube-scheduler. Understanding compute resource specifications, Quality of Service (QoS) classes, and node selection mechanics (nodeSelector, Node Affinity, Taints & Tolerations) is essential for managing cluster efficiency.

---

1. Resource Requests vs. Limits

Containers specify their compute resource needs using two attributes inside the container spec: requests and limits.

CPU & Memory Requests

• Requests represent the minimum guaranteed resources required by a container.
• The kube-scheduler uses the sum of resource requests on a node to evaluate node capacity during scheduling. A node is eligible to run a Pod only if its allocatable capacity exceeds the Pod's total resource requests.
• CPU Units: Expressed in millicores (m) or fractional cores (500m = 0.5 CPU core; 1000m = 1 full vCPU/core).
• Memory Units: Expressed in binary bytes using mebibytes (Mi) or gibibytes (Gi) (e.g., 256Mi, 2Gi).

CPU & Memory Limits

• Limits specify the maximum hard ceiling of resources a container is permitted to consume.
• CPU Limit Behavior: Enforced via Linux cgroup Completely Fair Scheduler (CFS) quota. If a container exceeds its CPU limit, the OS throttles the container's CPU usage, causing performance slowdowns without terminating the process.
• Memory Limit Behavior: Enforced via Linux cgroups. If a container exceeds its memory limit, the operating system kernel immediately invokes the Out-Of-Memory (OOM) killer, terminating the container process. The Pod enters an OOMKilled state with exit code 137 and is restarted by the Kubelet.

resources:
  requests:
    cpu: "250m"
    memory: "256Mi"
  limits:
    cpu: "500m"
    memory: "512Mi"

---

2. Quality of Service (QoS) Classes

When worker nodes experience memory exhaustion, Kubernetes must evict Pods to protect host stability. Kubernetes automatically classifies Pods into three Quality of Service (QoS) classes based on their resource requests and limits configuration:

1. Guaranteed

• Criteria: Every container in the Pod must have both CPU and memory requests and limits explicitly specified, and the request value must equal the limit value for every resource.
• Eviction Priority: Lowest eviction priority. Guaranteed Pods are only evicted if node memory pressure is extreme and no lower-tier Pods remain.

2. Burstable

• Criteria: The Pod does not meet Guaranteed criteria, but at least one container in the Pod has a CPU or memory request specified.
• Eviction Priority: Intermediate eviction priority. Burstable Pods can consume extra unallocated node resources up to their limit, but will be evicted before Guaranteed Pods if node memory runs low.

3. BestEffort

• Criteria: No containers in the Pod have any CPU or memory requests or limits defined.
• Eviction Priority: Highest eviction priority. BestEffort Pods are the very first candidates selected by the Kubelet for termination when a node encounters memory pressure.

---

3. Node Selection Mechanics

By default, kube-scheduler distributes Pods evenly across healthy nodes. However, workloads often require specific hardware characteristics (such as SSD storage, GPU access, or specific availability zones).

nodeSelector

The simplest node selection mechanism is nodeSelector. It is a hard constraint key-value map placed in the Pod spec that matches node labels.

spec:
  nodeSelector:
    disktype: ssd
    topology.kubernetes.io/zone: us-east-1a

Node Affinity and Anti-Affinity

Node Affinity extends nodeSelector with a more expressive expression syntax, supporting logical operators (In, NotIn, Exists, DoesNotExist, Gt, Lt).

Node affinity supports two rules:

• requiredDuringSchedulingIgnoredDuringExecution: Hard rule. The scheduler must find a node matching the affinity rule, or the Pod remains Pending.
• preferredDuringSchedulingIgnoredDuringExecution: Soft rule. The scheduler attempts to place the Pod on a matching node, but can fall back to another node if no match is available. Soft rules include a weight (1–100) to prioritize nodes.

[!NOTE] Notice the phrase IgnoredDuringExecution: If node labels change after a Pod is already scheduled and running, the Pod will continue running uninterrupted.

---

4. Taints and Tolerations

While Node Affinity attracts Pods to specific nodes, Taints and Tolerations allow nodes to repel Pods.

• Taints are applied to Nodes (key=value:effect). A tainted node refuses any Pod that does not possess a matching toleration.
• Tolerations are applied to Pods, allowing (but not forcing) those Pods to be scheduled onto nodes with matching taints.

Taint Effects

1. NoSchedule: The scheduler will not place new Pods onto the node unless the Pod has a matching toleration. Existing running Pods are not affected.
2. PreferNoSchedule: The scheduler tries to avoid placing non-tolerating Pods on the node, but can place them there if no other node capacity exists.
3. NoExecute: New Pods without matching tolerations are blocked from scheduling, and existing running Pods on the node lacking the toleration are immediately evicted.

# Command to taint a node:
# kubectl taint nodes node1 dedicated=gpu:NoSchedule

apiVersion: v1
kind: Pod
metadata:
  name: gpu-workload
spec:
  containers:
  - name: cuda-app
    image: nvidia/cuda:12.0
  tolerations:
  - key: "dedicated"
    operator: "Equal"
    value: "gpu"
    effect: "NoSchedule"
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: gpu-type
            operator: In
            values: ["nvidia-a100"]

---

5. QoS & Scheduling Feature Summary