4.1 Kubernetes IP-per-Pod Model & CNI Plugins (Flannel, Calico, Cilium)

4.1 Kubernetes IP-per-Pod Model & CNI Plugins (Flannel, Calico, Cilium)

Networking is one of the most heavily tested domains on the Certified Kubernetes Administrator (CKA) examination. Kubernetes departs fundamentally from legacy container networking models (such as Docker's default host-port mapping and host-private NAT bridges) by establishing a clean, uniform, cluster-wide flat network abstraction known as the IP-per-Pod model.

To implement this model across heterogeneous infrastructure (bare-metal servers, virtual machines, public cloud VPCs), Kubernetes decouples the container runtime from network implementation details via the Container Network Interface (CNI) specification. Understanding how CNI plugins allocate IP addresses, wire virtual network interfaces, route inter-node traffic, and enforce security boundaries is essential for any production Kubernetes administrator.

1. The Kubernetes Network Model Requirements

The Kubernetes network specification establishes four foundational invariants that every cluster implementation must satisfy:

1. All Pods can communicate with all other Pods on any node without Network Address Translation (NAT).
2. All agents on a node (such as the kubelet and system daemons) can communicate with all Pods running on that same node.
3. The IP address that a Pod sees as its own is the exact same IP address that every other Pod and host in the cluster sees it as (no masquerading or port mapping).
4. Pods configured with hostNetwork: true can communicate with all Pods across all nodes within the network namespace of the host.

+-----------------------------------------------------------------------------------------+
|                         KUBERNETES IP-PER-POD NETWORK MODEL                             |
|                                                                                         |
|   WORKER NODE 01 (Host IP: 192.168.1.10)      WORKER NODE 02 (Host IP: 192.168.1.20)    |
|   Pod CIDR: 10.244.1.0/24                     Pod CIDR: 10.244.2.0/24                   |
|   +------------------------------------+      +------------------------------------+    |
|   |  +--------------+  +-------------+ |      |  +--------------+  +-------------+ |    |
|   |  | Pod A        |  | Pod B       | |      |  | Pod C        |  | Pod D       | |    |
|   |  | 10.244.1.2   |  | 10.244.1.3  | |      |  | 10.244.2.2   |  | 10.244.2.3  | |    |
|   |  +-------+------+  +------+------+ |      |  +-------+------+  +------+------+ |    |
|   |          | (veth)         | (veth) |      |          | (veth)         | (veth) |    |
|   |          v                v        |      |          v                v        |    |
|   |       [ cni0 / CNI Data Path ]     |      |       [ cni0 / CNI Data Path ]     |    |
|   +------------------+-----------------+      +------------------+-----------------+    |
|                      |                                           |                      |
|                      +================== UNDERLAY ================+                      |
|                                (No NAT / Routable or Overlay)                           |
+-----------------------------------------------------------------------------------------+

Architectural Advantages of IP-per-Pod:

• No Port Collision: Two applications running on port 8080 (e.g., microservices) can run concurrently on the same worker node without requiring port remapping or dynamic host port allocation.
• Clean Service Discovery: DNS records directly map service names to stable virtual IPs (VIPs) or individual Pod IPs.
• Simplified Telemetry & Logging: Network traces, firewall logs, and packet captures retain genuine, un-NATed IP addresses.

2. The Container Network Interface (CNI) Architecture

The Container Network Interface (CNI) is a CNCF project that standardizes how container runtimes (like containerd and CRI-O) interact with network plugins to configure container network namespaces.

+-----------------------------------------------------------------------------------------+
|                           CNI PLUGIN EXECUTION LIFECYCLE                                |
|                                                                                         |
|  [1. Kubelet / CRI Runtime]                                                             |
|         | (Spawns Pod sandbox & creates Linux network namespace: /var/run/netns/<id>)   |
|         v                                                                               |
|  [2. Read CNI Configs] ---> Inspects /etc/cni/net.d/ (Sorted Lexicographically)         |
|         |                   (e.g., 10-calico.conflist, 10-flannel.conflist)             |
|         v                                                                               |
|  [3. Invoke CNI Binary] ---> Calls executable in /opt/cni/bin/ (e.g., calico, flannel)  |
|         |                    Environment: CNI_COMMAND=ADD, CNI_CONTAINERID=...,         |
|         |                                 CNI_NETNS=/var/run/netns/..., CNI_IFNAME=eth0 |
|         v                                                                               |
|  [4. IPAM & Veth Wiring]---> Calls IPAM plugin (host-local / calico-ipam) for Pod IP    |
|         |                    Creates virtual ethernet pair (vethxxxx <-> eth0)          |
|         |                    Plumbs eth0 into Pod netns, sets default gateway & routes  |
|         v                                                                               |
|  [5. JSON Response]    ---> Returns assigned IP, MAC, routes back to Kubelet in JSON   |
+-----------------------------------------------------------------------------------------+

CNI Directory Layout & Configuration:

• Configuration Directory (/etc/cni/net.d/): Contains JSON or .conflist configuration files. The container runtime reads these files in lexicographical order; the first file in alphabetical order defines the primary CNI plugin.
• Binary Directory (/opt/cni/bin/): Contains standalone executable binaries invoked by the runtime, such as bridge, loopback, host-local, calico, flannel, cilium-cni, portmap, and macvlan.

Core CNI Operations:

1. ADD: Invoked when a Pod sandbox is created. Creates the virtual interface inside the Pod namespace, assigns an IP address from the IPAM pool, sets up default routes, and connects the host-side veth interface to the host network.
2. DEL: Invoked when a Pod is terminated. Deallocates the Pod IP address back to the IPAM pool, deletes the veth pair, and cleans up routing/iptables entries.
3. CHECK: Probes an existing container's network namespace to verify that expected interfaces, routes, and IP allocations remain valid.
4. VERSION: Queries the plugin binary for supported CNI specification versions.

[!IMPORTANT] The NetworkPluginNotReady Node Condition: When a worker node boots or joins a cluster, the kubelet remains in NotReady status with the message container runtime network not ready: NetworkPluginNotReady until a valid CNI configuration file is placed in /etc/cni/net.d/ and the backing CNI plugin daemonset starts successfully.

3. Deep Architectural Comparison: Flannel vs. Calico vs. Cilium

Production Kubernetes environments deploy different CNI implementations depending on performance requirements, underlying cloud fabric capabilities, and security policy requirements.

1. Flannel Architecture

Flannel (developed originally by CoreOS) is designed for simplicity. It allocates a subnet (e.g., a /24 from the cluster /16 Pod CIDR) to each node via etcd or the Kubernetes API (spec.podCIDR).

• VXLAN Backend (Default): Flannel creates a virtual interface (flannel.1) on each node. Outgoing Pod packets destined for a remote node are encapsulated inside a UDP packet (port 4789) by the Linux kernel VXLAN driver, transmitted over the physical underlay, and decapsulated by the remote node's flannel.1 device.
• Host-GW Backend: If all nodes reside on the same Layer 2 physical network broadcast domain, Flannel configures direct static routes in the host kernel routing table (using the destination node IP as the next-hop gateway), eliminating overlay encapsulation overhead entirely.

[!CAUTION] Flannel Does NOT Enforce NetworkPolicies: Flannel is strictly a packet transport plugin. If you deploy a Kubernetes NetworkPolicy resource on a cluster running standard Flannel CNI, all network traffic will continue to be allowed. The policies are silently ignored by the data plane unless an auxiliary policy controller (like Canal, which pairs Flannel routing with Calico policy enforcement) is installed.

2. Project Calico Architecture

Calico supports multiple data planes: unencapsulated Layer 3 routing with BGP as well as IP-in-IP or VXLAN overlays. The selected mode depends on underlay routing and operator configuration; do not assume every Calico installation advertises Pod routes directly.

• Felix: The core Calico node agent running as a DaemonSet. Programs Linux kernel routing tables, iptables chains, and eBPF maps to enforce NetworkPolicies and route packets to local veth interfaces.
• BIRD (BGP Daemon): Runs on every node to peer with other nodes (Node-to-Node Mesh) or with Top-of-Rack (ToR) physical switches (BGP Route Reflectors), advertising host /26 or /24 Pod IP blocks.
• IPAM (calico-ipam): Allocates dynamic IP blocks (IPPool) with support for cross-subnet CIDR allocation and CIDR auto-detection.
• Encapsulation Options (for routed networks): If intermediate physical routers drop unknown Pod IPs, Calico can be configured to use IP-in-IP (protocol 4) or VXLAN (UDP port 4789) encapsulation selectively across subnet boundaries.

3. Cilium Architecture

Cilium is an advanced, high-performance CNI plugin engineered entirely around eBPF (Extended Berkeley Packet Filter).

• Kernel Bypass: Traditional packet forwarding requires traversing lengthy Netfilter/iptables chains inside the Linux kernel. Cilium attaches eBPF bytecode programs directly to the network device drivers (using XDP - eXpress Data Path) and Linux socket layer (sock_ops).
• Kube-Proxy Replacement: Cilium handles Kubernetes Services directly in eBPF maps at socket creation time, replacing kube-proxy and eliminating iptables/IPVS rule table bloat.
• L7 Network Security: Cilium can inspect and filter HTTP methods, URL paths, headers, and gRPC methods directly in the data path.
• Hubble: Built-in network observability platform providing real-time graphical visibility into service dependencies, packet drops, DNS resolution latencies, and TLS handshakes.

4. MTU Tuning & Encapsulation Overhead

A frequent source of mysterious network degradation, intermittent connection timeouts, and SSL/TLS handshake freezes in Kubernetes clusters is MTU (Maximum Transmission Unit) misconfiguration.

+-----------------------------------------------------------------------------------------+
|                         PACKET ENCAPSULATION & MTU OVERHEAD                             |
|                                                                                         |
|   Standard Ethernet Frame (MTU: 1500 Bytes)                                             |
|   +---------------------------------------------------------------------------------+   |
|   | Standard IP Header (20B) | TCP Header (20B) | TCP Payload / Data (1460 Bytes)   |   |
|   +---------------------------------------------------------------------------------+   |
|                                                                                         |
|   VXLAN Encapsulated Frame (Requires 50 Bytes of Overhead)                              |
|   +---------------------------------------------------------------------------------+   |
|   | Outer IP (20B) | UDP (8B) | VXLAN (8B) | Inner MAC (14B) | Inner IP | TCP | Data|   |
|   +---------------------------------------------------------------------------------+   |
|   |<--------------------------- 50 Bytes Overhead ---------------->|                    |
|   Effective Inner Pod MTU Must Be: 1500 - 50 = 1450 Bytes                               |
+-----------------------------------------------------------------------------------------+

Encapsulation Overhead Calculation:

• Standard Physical Underlay MTU: Typically 1500 bytes (or 9000 bytes for Jumbo Frames).
• VXLAN Overhead: 50 bytes (Outer IP: 20B + UDP: 8B + VXLAN: 8B + Inner Ethernet Frame: 14B). The Pod network interface MTU must be set to $\le 1450$.
• Geneve Overhead: 50 bytes. Pod MTU must be $\le 1450$.
• IP-in-IP Overhead: 20 bytes (Outer IP header). Pod MTU must be $\le 1480$.
• WireGuard Encryption Overhead: 60 to 80 bytes depending on underlay encapsulation.

[!TIP] Symptom of MTU Mismatch: Small packets (such as ping ICMP requests and TCP SYN packets) succeed because they fit within the MTU, but large data transfers (such as curl downloading a large JSON payload or TLS certificate exchange) hang indefinitely due to unhandled packet fragmentation.

5. CNI Inspection & Troubleshooting Runbook

When diagnosing networking issues during the CKA examination, follow this systematic command runbook:

# 1. Inspect CNI configuration directory
ls -la /etc/cni/net.d/
cat /etc/cni/net.d/10-calico.conflist

# 2. Verify CNI plugin binaries are installed
ls -la /opt/cni/bin/

# 3. Check CNI DaemonSet pods in kube-system
kubectl get pods -n kube-system -o wide -l k8s-app=calico-node
kubectl get pods -n kube-system -o wide -l app=flannel
kubectl get pods -n kube-system -o wide -l k8s-app=cilium

# 4. View CNI agent logs on a failing worker node
kubectl logs -n kube-system daemonset/calico-node -c calico-node --tail=100

# 5. Inspect host network interfaces and routing tables
ip link show
ip route show

# 6. Verify Pod IP address assignment and veth binding
kubectl get pods -o wide
ip netns list