2.3 Container Orchestration & Kubernetes Architecture

Container Orchestration & Kubernetes Architecture

As cloud-native architectures scale from dozens to tens of thousands of container instances across multi-region environments, manual deployment, static port mapping, and individual host management become impossible. Container orchestration platforms automate the deployment, scaling, health monitoring, service discovery, traffic routing, rolling upgrades, and storage management of containerized workloads across clusters of physical or virtual servers.

Kubernetes (K8s) is the industry-standard container orchestrator. Mastering Kubernetes architectural components, workload primitives, networking models, and storage integration is a critical domain on the CompTIA Cloud+ (CV0-004) exam.

---

1. Kubernetes Control Plane vs. Worker Node Architecture

A Kubernetes cluster is partitioned into two distinct operational planes: the Control Plane (which makes global cluster decisions, detects events, and responds to desired state changes) and Worker Nodes (which execute the actual application container workloads).

+-----------------------------------------------------------------------------+
|                     KUBERNETES CLUSTER ARCHITECTURE                         |
|                                                                             |
|   +---------------------------------------------------------------------+   |
|   |                   CONTROL PLANE (MASTER NODES)                      |   |
|   |                                                                     |   |
|   |   +-------------------------------------------------------------+   |   |
|   |   |                     kube-apiserver                          |   |   |
|   |   | (Central REST API Gateway, Authentication & Admission Ctrl) |   |   |
|   |   +-------------------------------------------------------------+   |   |
|   |        |                     |                     |                |   |
|   |        v                     v                     v                |   |
|   |   +----------+      +------------------+  +--------------------+    |   |
|   |   |   etcd   |      |  kube-scheduler  |  | kube-controller-   |    |   |
|   |   | (Raft KV |      | (Resource Sizing |  |      manager       |    |   |
|   |   |  Store)  |      |  Node Placement) |  |  (Reconciliation)  |    |   |
|   |   +----------+      +------------------+  +--------------------+    |   |
|   +---------------------------------------------------------------------+   |
|                                     |                                       |
|                  +------------------+------------------+                    |
|                  | (TLS Encrypted Communication)       |                    |
|                  v                                     v                    |
|   +-----------------------------+       +-----------------------------+     |
|   |        WORKER NODE 1        |       |        WORKER NODE 2        |     |
|   |  +-----------------------+  |       |  +-----------------------+  |     |
|   |  | kubelet               |  |       |  | kubelet               |  |     |
|   |  | (Node Agent & PodSpec)|  |       |  | (Node Agent & PodSpec)|  |     |
|   |  +-----------------------+  |       |  +-----------------------+  |     |
|   |  | kube-proxy            |  |       |  | kube-proxy            |  |     |
|   |  | (iptables/IPVS Rules) |  |       |  | (iptables/IPVS Rules) |  |     |
|   |  +-----------------------+  |       |  +-----------------------+  |     |
|   |  | Container Runtime     |  |       |  | Container Runtime     |  |     |
|   |  | (containerd / CRI-O)  |  |       |  | (containerd / CRI-O)  |  |     |
|   |  +-----------------------+  |       |  +-----------------------+  |     |
|   |  | [Pod 1]  [Pod 2]      |  |       |  | [Pod 3]  [Pod 4]      |  |     |
|   |  +-----------------------+  |       |  +-----------------------+  |     |
|   +-----------------------------+       +-----------------------------+     |
+-----------------------------------------------------------------------------+

Control Plane Components Deep Dive

1. kube-apiserver:
   • The central nervous system of Kubernetes. It exposes the Kubernetes REST API (HTTPS port 6443) and serves as the front-end for all cluster communications.
   • Every administrative command (kubectl), worker node daemon (kubelet), and internal controller interacts exclusively with the API server. No component ever interacts directly with etcd except the kube-apiserver.
   • Enforces authentication (client certificates, bearer tokens, OpenID Connect), authorization (RBAC - Role-Based Access Control, ABAC), and schema validation via Admission Controllers.
2. etcd (Distributed Key-Value Store):
   • A consistent, highly available, distributed key-value store that holds the complete canonical state, configuration parameters, and metadata of the entire Kubernetes cluster.
   • Operates on the Raft consensus algorithm to ensure strong data consistency. In production, etcd is deployed in high-availability clusters with an odd number of nodes (3, 5, or 7 members) to maintain quorum during network partitions ($Quorum = \lfloor N/2 \rfloor + 1$).
3. kube-scheduler:
   • Watches for newly created Pods that have no assigned worker node (spec.nodeName is blank).
   • Evaluates node resource capacity, compute requests/limits, hardware constraints, node affinity/anti-affinity rules, taints and tolerations, and topology spread constraints to select the optimal worker node for each Pod.
4. kube-controller-manager:
   • Runs core background reconciliation loops that continuously compare the current actual state of the cluster against the desired state defined in manifests, executing corrective actions when deviations occur.
   • Bundles key controllers: Node Controller (tracks node health), ReplicaSet Controller (maintains replica counts), EndpointSlice Controller (joins Pods to Services), and ServiceAccount Controller.
5. cloud-controller-manager:
   • Interfaces directly with underlying cloud service provider APIs (AWS, Azure, GCP) to dynamically provision cloud-native resources such as external Cloud Load Balancers, VPC route tables, and cloud persistent disks.

Worker Node Daemons

1. kubelet:
   • The primary node agent running on every worker node. It registers the physical/virtual node with the kube-apiserver.
   • Watches for Pod assignments from the API server, downloads the PodSpec, interacts with the local container runtime via the Container Runtime Interface (CRI) to start/stop containers, mounts persistent storage volumes, and executes container Liveness, Readiness, and Startup probes.
2. kube-proxy:
   • The network proxy running on each node that implements Kubernetes Service abstractions.
   • Maintains host-level firewall rules (iptables or IPVS - IP Virtual Server) to intercept traffic directed to Service ClusterIPs and load-balance connections across backend Pod endpoints.
3. Container Runtime Engine:
   • The software responsible for running containers (e.g., containerd, CRI-O). It implements the OCI specifications and executes low-level runtimes (runc).

---

2. Core Kubernetes Workload Primitives & Lifecycle

Kubernetes workloads are defined declaratively in YAML or JSON manifests. Understanding workload primitives and choosing the appropriate controller is essential for cloud architecture.

+-----------------------------------------------------------------------------+
|                       KUBERNETES WORKLOAD PRIMITIVES                        |
|                                                                             |
|   POD                deployments               statefulsets                 |
|   +--------------+   +---------------------+   +--------------------------+ |
|   | Atomic unit  |   | Stateless apps      |   | Stateful databases       | |
|   | 1+ containers|   | Rolling updates     |   | Stable network names:    | |
|   | Shared IP/Net|   | Replicas scaling    |   | (db-0, db-1, db-2)       | |
|   | Shared Vol   |   | Automated rollback  |   | Ordered deployment       | |
|   +--------------+   +---------------------+   +--------------------------+ |
|                                                                             |
|   daemonsets                           jobs & cronjobs                      |
|   +--------------------------------+   +----------------------------------+ |
|   | Exactly 1 Pod per node         |   | Run-to-completion batch processes| |
|   | (Log collectors, monitoring,   |   | Jobs: One-off database migration | |
|   | CNI networking daemons)        |   | CronJobs: Scheduled tasks        | |
|   +--------------------------------+   +----------------------------------+ |
+-----------------------------------------------------------------------------+

1. Pods (Atomic Execution Unit)

A Pod is the smallest and simplest unit in the Kubernetes object model. A Pod represents a single instance of a running process in the cluster.

• Multi-Container Pod Patterns: While most pods contain a single container, multi-container pods share the exact same Network namespace (can communicate over localhost and share the same IP) and storage volumes:
  • Sidecar Pattern: Enhances or extends the main application container (e.g., an Envoy proxy sidecar for mTLS, or a Fluentbit sidecar streaming log files).
  • Init Container Pattern: Specialized containers that run sequentially to completion before app containers start (e.g., executing schema migrations or waiting for a database port to become reachable).
  • Ambassador Pattern: Proxies connections from the main container to external worlds (e.g., abstracting local database connections to a cloud database cluster).

2. Workload Controllers

• Deployments & ReplicaSets:
  • Designed for stateless applications (e.g., web frontends, stateless REST APIs).
  • The Deployment manages an underlying ReplicaSet, which guarantees that a specific number of identical Pod replicas are running at all times.
  • Supports declarative update strategies: RollingUpdate (incrementally replaces old pods with new pods with zero downtime) and Recreate (terminates all old pods before launching new ones).
• StatefulSets:
  • Designed for stateful applications requiring persistent identity (e.g., PostgreSQL, MongoDB, Apache Kafka, Elasticsearch).
  • Guarantees: Stable, predictable network hostnames (db-0, db-1, db-2), ordered graceful deployment and termination (starts 0, then 1, then 2), and automated binding to dedicated PersistentVolumeClaims per ordinal index via volumeClaimTemplates.
• DaemonSets:
  • Guarantees that a copy of a specific Pod runs on all (or selected) worker nodes in the cluster. As new nodes are added to the cluster, the DaemonSet automatically schedules pods onto them.
  • Common Use Cases: Cluster logging agents (Fluentd, Promtail), host monitoring exporters (Prometheus Node Exporter), and CNI network daemons (Calico/Cilium).
• Jobs & CronJobs:
  • Job: Creates one or more pods and ensures that a specified number of them successfully terminate upon completing a finite task (e.g., data transformation, backup execution).
  • CronJob: Manages Jobs that execute on a repeating time-based schedule using standard crontab syntax (e.g., 0 2 * * * for nightly execution).

---

3. Kubernetes Networking & Service Routing

The Kubernetes network model is fundamentally flat and operates on four foundational rules:

1. All Pods can communicate with all other Pods on any node without Network Address Translation (NAT).
2. All Node agents (kubelet) can communicate with all Pods on that node.
3. Every Pod receives its own unique, routable IP address from the cluster Pod CIDR range.
4. IP addresses assigned to Pods are dynamic and ephemeral (destroyed when a pod restarts).

+-----------------------------------------------------------------------------+
|                        KUBERNETES SERVICE TYPES                             |
|                                                                             |
|   1. ClusterIP (Default: Internal cluster-only communication)               |
|      [Pod A] ---> [ClusterIP VIP: 10.96.0.10] ---> [Pod B1] or [Pod B2]     |
|                                                                             |
|   2. NodePort (Static port 30000-32767 exposed on every worker node)        |
|      [Client] ---> [NodeIP:31234] ---> [ClusterIP] ---> [Pod Endpoints]     |
|                                                                             |
|   3. LoadBalancer (Provisions external cloud provider Load Balancer)        |
|      [Client] ---> [AWS NLB / Azure LB: 52.1.2.3] ---> [NodePorts] ---> Pods|
|                                                                             |
|   4. ExternalName (Internal DNS CNAME alias to external service)            |
|      [Pod] ---> [my-db.svc] ---> CNAME ---> [db.production.rds.aws.com]     |
+-----------------------------------------------------------------------------+

Service Types Breakdown:

• ClusterIP (Default): Exposes the Service on an internal, non-routable virtual IP (VIP) accessible only from within the Kubernetes cluster. Ideal for internal microservices, backend databases, and caching layers.
• NodePort: Allocates a dedicated port from a reserved cluster-wide range (default: 30000–32767) across every worker node's physical/virtual IP address. External traffic hitting <Any-Node-IP>:<NodePort> is forwarded to the Service.
• LoadBalancer: Integrates with the cloud provider's cloud-controller-manager to automatically provision an enterprise external load balancer (e.g., AWS Network Load Balancer, Azure Standard Load Balancer, GCP Cloud Load Balancing) that routes public traffic directly to the service's NodePorts.
• ExternalName: Maps an internal Kubernetes service name directly to an external DNS CNAME record (e.g., pointing order-db.production.svc to an external AWS RDS endpoint) without proxying traffic.
• Headless Service (spec.clusterIP: None): Does not allocate a virtual ClusterIP. Instead, internal CoreDNS queries return the direct A records (IP addresses) of all backing individual Pods, enabling peer discovery for stateful distributed databases.

Ingress Controllers & Gateway API (Layer 7 Routing)

While a LoadBalancer service creates a separate, expensive Layer 4 load balancer for every individual application, an Ingress Controller provides intelligent Layer 7 HTTP/HTTPS routing through a single public IP.

# Production Ingress Resource Definition
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: production-routing
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - api.example.com
    secretName: api-example-tls
  rules:
  - host: api.example.com
    http:
      paths:
      - path: /v1/orders
        pathType: Prefix
        backend:
          service:
            name: order-service
            port:
              number: 8080
      - path: /v1/users
        pathType: Prefix
        backend:
          service:
            name: user-service
            port:
              number: 8081

• Ingress Controller Daemons: Reverse proxies (e.g., NGINX Ingress, Traefik, HAProxy, AWS ALB Ingress Controller) that monitor Ingress resources via the API server and dynamically update routing tables, execute TLS termination, and perform URL rewrites.
• Kubernetes Gateway API: The modern, expressive evolution of Ingress that separates infrastructure management (GatewayClass), cluster operations (Gateway), and developer routing (HTTPRoute, GRPCRoute).

---

4. Container Network Interface (CNI) & Network Policies

The Container Network Interface (CNI) is a Cloud Native Computing Foundation (CNCF) project that defines a standardized plugin interface for configuring network connectivity and tearing down interfaces for Linux containers.

Major CNI Plugin Architectures:

• Flannel: A simple, lightweight overlay network using VXLAN encapsulation. Simple to configure, but does not support Kubernetes NetworkPolicies.
• Calico: Highly scalable, enterprise CNI utilizing pure Layer 3 Border Gateway Protocol (BGP) routing without packet encapsulation, delivering near-native network speed. Features an industry-standard policy engine for micro-segmentation.
• Cilium: Next-generation CNI powered by eBPF (Extended Berkeley Packet Filter) directly inside the Linux kernel. Provides high-performance packet routing, deep L7 visibility, and transparent encryption without iptables overhead.
• AWS VPC CNI / Azure CNI: Native cloud provider CNIs that assign real, routable VPC/VNet secondary private IP addresses directly to Kubernetes Pods, enabling direct integration with cloud Security Groups, Network Security Groups (NSGs), and AWS Direct Connect.

Kubernetes NetworkPolicies (Zero-Trust Micro-Segmentation)

By default, Kubernetes networking is non-isolated—any Pod in any namespace can establish network connections with any other Pod. NetworkPolicies act as stateful Layer 3/Layer 4 firewalls for Pods.

# Secure Default-Deny & Ingress Whitelist NetworkPolicy
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: secure-db-policy
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: postgres-db
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: order-service
    ports:
    - protocol: TCP
      port: 5432

[!IMPORTANT] NetworkPolicy Enforcement Requirement: Defining a NetworkPolicy YAML object does nothing unless the cluster is running a CNI plugin that actively enforces policies (e.g., Calico, Cilium, or Weave Net). If a cluster runs Flannel alone, NetworkPolicy resources are ignored, leaving network traffic completely unsegmented.

---

5. Container Storage Interface (CSI), PVs, PVCs & StorageClasses

Containers are ephemeral by default. To attach durable enterprise block, file, or object storage to Pods, Kubernetes utilizes the Container Storage Interface (CSI).

+-----------------------------------------------------------------------------+
|                   DYNAMIC STORAGE PROVISIONING WORKFLOW                     |
|                                                                             |
|   1. CLUSTER ADMIN DEFINES STORAGECLASS                                     |
|      [StorageClass: 'gp3-sc'] ---> Provisioner: ebs.csi.aws.com             |
|                                                                             |
|   2. DEVELOPER CREATES PERSISTENTVOLUMECLAIM (PVC)                          |
|      [PVC: 'db-pvc'] ---> Requests 100Gi, ReadWriteOnce, storageClass: gp3  |
|                                                                             |
|   3. CSI PROVISIONER AUTOMATICALLY PROVISIONS PHYSICAL VOLUME               |
|      [CSI Controller Plugin] ---> Calls AWS API ---> Creates 100Gi EBS Vol  |
|                                                                             |
|   4. PV OBJECT CREATED & BOUND TO PVC                                       |
|      [PersistentVolume: 'pv-09af'] <========== BOUND ==========> [PVC]     |
|                                                                             |
|   5. POD MOUNTS VOLUME INTO CONTAINER PATH                                  |
|      [Stateful Pod] ---> volumeMounts: /var/lib/postgresql/data             |
+-----------------------------------------------------------------------------+

Core Storage Objects:

• StorageClass: Defines the dynamic storage provisioner (e.g., ebs.csi.aws.com, disk.csi.azure.com), volume performance parameters (IOPS, throughput, disk type), and the reclaimPolicy (Delete to wipe the volume when PVC is deleted, or Retain to preserve volume for manual administrator recovery).
• PersistentVolumeClaim (PVC): A developer's declarative request for storage specifying capacity (e.g., 500Gi), storage class, and access mode.
• PersistentVolume (PV): The actual representation of physical cloud storage in the cluster, either pre-provisioned by an administrator or automatically provisioned by the CSI controller.

Storage Access Modes:

• ReadWriteOnce (RWO): Volume can be mounted as read-write by a single node only. Standard for Cloud Block Storage (AWS EBS, Azure Managed Disk, GCP Persistent Disk).
• ReadOnlyMany (ROX): Volume can be mounted as read-only by multiple nodes concurrently.
• ReadWriteMany (RWX): Volume can be mounted as read-write by multiple nodes simultaneously. Requires Cloud File Storage (NFS, AWS EFS, Azure Files) or distributed filesystems (Ceph, GlusterFS).
• ReadWriteOncePod (RWOP): Volume can be mounted as read-write by a single Pod across the entire cluster (introduced in Kubernetes 1.22+ for strict single-writer block storage access).

---

6. CompTIA Cloud+ Exam Tips & Common Pod Failures

• CrashLoopBackOff: The container process repeatedly starts, encounters an error (missing environment variable, invalid database password, unhandled exception), and crashes. Troubleshoot with kubectl logs <pod-name> --previous and kubectl describe pod <pod-name>.
• ImagePullBackOff / ErrImagePull: The worker node cannot pull the specified container image. Root causes: Typo in image repository tag, private registry authentication failure (missing imagePullSecrets), or network routing/firewall blocking registry access.
• Pending Status: The Pod cannot be scheduled onto any worker node. Root causes: Insufficient cluster CPU/memory capacity (ResourceQuota reached), node selector/affinity mismatch, or no nodes tolerating a specific taint.
• Multi-Attach error for volume: Occurs when a StatefulSet or Deployment with an RWO (ReadWriteOnce) block volume is rescheduled onto a new worker node, but the cloud provider has not released the volume lock from the old node.