3.3 DaemonSets & StatefulSets Architecture

DaemonSets & StatefulSets Architecture

While Deployments manage stateless, interchangeable Pods, specialized workloads require distinct scheduling and identity guarantees. DaemonSets guarantee node-level infrastructure services (networking, logging, monitoring), while StatefulSets provide stable network identities, ordered rollouts, and persistent storage bindings for stateful distributed datastores (e.g., PostgreSQL, Kafka, Cassandra, ZooKeeper).

---

1. DaemonSet Architecture and Scheduling Mechanics

A DaemonSet ensures that all (or a subset of) nodes run exactly one copy of a Pod. As new nodes are added to the cluster, the DaemonSet controller automatically schedules the DaemonSet Pod onto them; as nodes are removed, those Pods are garbage collected.

+-----------------------------------------------------------------------------------------+
|                                DAEMONSET ARCHITECTURE                                   |
|                                                                                         |
|                              +-----------------------+                                  |
|                              | DaemonSet Controller  |                                  |
|                              | (e.g., fluentd-logger)|                                  |
|                              +-----------------------+                                  |
|                                          |                                              |
|             +----------------------------+----------------------------+                 |
|             |                            |                            |                 |
|             v                            v                            v                 |
|     +---------------+            +---------------+            +---------------+         |
|     |    Node 1     |            |    Node 2     |            |    Node 3     |         |
|     | +-----------+ |            | +-----------+ |            | +-----------+ |         |
|     | |DS Pod (1) | |            | |DS Pod (2) | |            | |DS Pod (3) | |         |
|     | +-----------+ |            | +-----------+ |            | +-----------+ |         |
|     +---------------+            +---------------+            +---------------+         |
+-----------------------------------------------------------------------------------------+

Primary Use Cases for DaemonSets

1. Cluster Network Routing (CNI): kube-flannel-ds, calico-node, cilium.
2. Cluster Log Collection: fluentd, fluent-bit, promtail reading /var/log on host nodes.
3. Node Metrics & Hardware Monitoring: node-exporter, NVIDIA GPU device plugins.
4. Storage Daemons: ceph-osd, glusterd.

Modern DaemonSet Scheduling

In modern Kubernetes, DaemonSet Pods are scheduled by the standard kube-scheduler rather than the DaemonSet controller directly. The controller injects default Node Affinity terms (spec.affinity.nodeAffinity) matching kubernetes.io/hostname and automatically appends tolerations for built-in taints:

• node.kubernetes.io/not-ready:NoExecute
• node.kubernetes.io/unreachable:NoExecute
• node.kubernetes.io/disk-pressure:NoSchedule
• node.kubernetes.io/memory-pressure:NoSchedule
• node.kubernetes.io/unschedulable:NoSchedule

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd-logging
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: fluentd-logger
  template:
    metadata:
      labels:
        name: fluentd-logger
    spec:
      tolerations:
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
      containers:
        - name: fluentd
          image: fluent/fluentd-kubernetes-daemonset:v1.16-debian-elasticsearch
          volumeMounts:
            - name: varlog
              mountPath: /var/log
      volumes:
        - name: varlog
          hostPath:
            path: /var/log

DaemonSet Update Strategies

• RollingUpdate (Default): Gradually terminates old DaemonSet Pods and creates new ones on each node. Supports maxUnavailable (default: 1) and maxSurge (Kubernetes 1.22+).
• OnDelete: New Pods are only created when the old DaemonSet Pod on a node is manually deleted.

---

2. StatefulSet Architecture and Guarantees

StatefulSets manage stateful applications that require one or more of the following:

1. Stable, unique network identifiers.
2. Stable, persistent storage.
3. Ordered, graceful deployment and scaling.
4. Ordered, automated rolling updates.

+-----------------------------------------------------------------------------------------+
|                                STATEFULSET ARCHITECTURE                                 |
|                                                                                         |
|   StatefulSet: web (replicas: 3)                                                        |
|   Headless Service: nginx (clusterIP: None)                                             |
|                                                                                         |
|       [web-0] --------------------> Bound to PVC: [data-web-0] (PV-001)                 |
|       DNS: web-0.nginx.default.svc.cluster.local                                        |
|          |                                                                              |
|          v (Must be Ready before web-1 starts)                                          |
|       [web-1] --------------------> Bound to PVC: [data-web-1] (PV-002)                 |
|       DNS: web-1.nginx.default.svc.cluster.local                                        |
|          |                                                                              |
|          v (Must be Ready before web-2 starts)                                          |
|       [web-2] --------------------> Bound to PVC: [data-web-2] (PV-003)                 |
|       DNS: web-2.nginx.default.svc.cluster.local                                        |
+-----------------------------------------------------------------------------------------+

Ordinal Indexing

For a StatefulSet with $N$ replicas, each Pod is assigned an integer ordinal from $0$ through $N-1$. The Pod name is formatted as $(statefulset-name)-$(ordinal) (e.g., cassandra-0, cassandra-1, cassandra-2).

Headless Services (clusterIP: None)

A StatefulSet requires a Headless Service (spec.serviceName) to govern the network domain. Because clusterIP is set to None, kube-DNS creates direct A records pointing to individual Pod IPs rather than a single virtual service IP:

$(pod-name).$(service-name).$(namespace).svc.cluster.local

Example: db-0.mysql-nodes.prod.svc.cluster.local

---

3. Persistent Volume Claim Templates (volumeClaimTemplates)

Unlike Deployments where all Pods share the same PVC reference, a StatefulSet provisions a distinct PersistentVolumeClaim for each Pod replica using volumeClaimTemplates.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: datastore
spec:
  serviceName: datastore-svc
  replicas: 3
  selector:
    matchLabels:
      app: datastore
  template:
    metadata:
      labels:
        app: datastore
    spec:
      containers:
        - name: redis
          image: redis:7.2-alpine
          ports:
            - containerPort: 6379
              name: redis
          volumeMounts:
            - name: redis-data
              mountPath: /data
  volumeClaimTemplates:
    - metadata:
      name: redis-data
      spec:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: "premium-ssd"
        resources:
          requests:
            storage: 50Gi

Storage Lifecycle & Retention Policies

• Scale-Down Non-Deletion: When a StatefulSet is scaled down (e.g., from 3 to 1), Pods datastore-2 and datastore-1 are deleted, but redis-data-datastore-2 and redis-data-datastore-1 are NOT deleted. This prevents data loss.
• Pod Reattachment: If the StatefulSet is scaled back up to 3, the new datastore-1 and datastore-2 Pods automatically reattach to their existing corresponding PVCs.
• persistentVolumeClaimRetentionPolicy (Kubernetes 1.27+ GA): Allows administrators to configure whether PVCs are deleted or retained when the StatefulSet is deleted (whenDeleted) or scaled down (whenScaled).

---

4. Deployment and Scaling Semantics

StatefulSets enforce strict ordering rules governed by spec.podManagementPolicy:

OrderedReady (Default)

• Startup: Pods start sequentially from index $0$ to $N-1$. Pod $i$ will not be created until Pod $i-1$ is fully Running and Ready.
• Scale Down / Termination: Pods terminate in strict reverse order from $N-1$ down to $0$. Pod $i$ must be completely shut down before Pod $i-1$ is terminated.

Parallel

• Pods are created and scaled simultaneously without waiting for neighboring ordinals. Suitable for batch-style distributed compute tasks where order is irrelevant.

---

5. StatefulSet Partitioned Rolling Updates

StatefulSet RollingUpdate strategy supports canary testing via the partition parameter:

spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 2

• When partition: 2 is set on a 4-replica StatefulSet (web-0, web-1, web-2, web-3), only Pods with ordinal >= 2 (web-2 and web-3) are updated to the new template.
• Pods web-0 and web-1 continue running the old version.
• Setting partition: 0 updates all remaining Pods.