5.2 OKE Networking: VCN-Native Pod Networking vs. Flannel, Subnet Sizing & Load Balancers

10.2 OKE Networking: VCN-Native Pod Networking (OCI-VCN CNI) vs Flannel

Container Network Interface (CNI) Architecture in OKE

Networking represents one of the most consequential architectural decisions when provisioning an OKE cluster. The Container Network Interface (CNI) plugin dictates how IP addresses are assigned to pods, how packets traverse the cluster fabric, how microservices communicate with external cloud and on-premises resources, and how security policies are enforced at the network boundary.

OKE supports two networking architectures: OCI VCN-Native Pod Networking and flannel overlay networking. Key rules:

• The network type is chosen when the cluster is created, and you cannot change the CNI plugin afterward. All node pools in a cluster use the same plugin.
• The Console uses VCN-native networking by default (Kubernetes 1.22 or later); the API defaults to flannel unless you choose otherwise.
• Virtual node pools require VCN-native networking. Flannel works only with managed and self-managed nodes.
• Calico can be installed to enforce Kubernetes NetworkPolicy with either plugin, and Oracle recommends the Cilium CNI for very large clusters (more than 5,000 nodes).

Choosing between the models affects IP address consumption, routability, and how security rules are applied.

---

OCI-VCN CNI (VCN-Native Pod Networking) Architecture

In the OCI-VCN CNI plugin model, Kubernetes pods are first-class citizens of the OCI Virtual Cloud Network (VCN):

Direct VCN IP Allocation

Every pod is allocated a private IP address from a VCN subnet, the pod subnet, which must be private and regional. Worker nodes use a primary VNIC for node traffic and one or more secondary VNICs whose secondary private IPs are handed to pods. The worker node subnet and the pod subnet must be in the same VCN; in some cases they can even be the same subnet.

No Overlay Encapsulation

Because pod IP addresses belong natively to the VCN, packets travel across the physical OCI fabric without software overlay encapsulation (such as VXLAN or Geneve). Pods therefore avoid the CPU overhead and extra headers of overlay encapsulation and can use the VCN's standard 9,000-byte MTU for traffic inside the VCN.

Direct Hybrid & Peering Routability

Because pod IPs reside within the VCN's routable address space, pods can directly communicate with:

• Other VCN subnets and peered VCNs via Local Peering Gateways (LPG) or Dynamic Routing Gateways (DRG v2).
• On-premises data centers traversing OCI FastConnect or Site-to-Site IPsec VPN without requiring Network Address Translation (NAT) or Ingress proxy hops.
• Autonomous Databases and Base DB systems residing in private DB subnets.
• External corporate systems that require initiating inbound TCP connections directly to specific pod IP addresses.

Security Rules for Pods

With VCN-native networking you control pod traffic with NSGs (recommended) or security lists defined for the pod subnet of a node pool. Those rules apply to all pods on the worker nodes attached to that pod subnet, so put workloads with different security needs into different node pools and pod subnets. For rules between individual pods inside the cluster, add Kubernetes NetworkPolicy enforcement, for example with Calico. Because pods have real VCN addresses, VCN Flow Logs show traffic by pod IP. Note that pod IPs are not persistent: a recreated pod usually gets a new address, so on-premises systems should target a Service or load balancer rather than a pod IP.

In clusters that use the VCN-native plugin, managed and self-managed nodes can also carry Zero Trust Packet Routing (ZPR) security attributes, which adds intent-based policy on top of NSGs and security lists (ZPR is not supported for virtual node pools or flannel clusters).

---

Flannel Overlay Network Architecture

Flannel is a mature, host-local overlay networking plugin that creates an isolated virtual network on top of the OCI VCN:

VXLAN Encapsulation

Flannel assigns each worker node a slice of an internal, cluster-wide private CIDR block (default 10.244.0.0/16). Pods receive IP addresses from this overlay range. When a pod communicates with a pod on another worker node, the Flannel daemon encapsulates the inner Ethernet frame inside a standard UDP packet (port 8472) on the sending node and decapsulates it on the receiving node.

IP Address Conservation

The primary advantage of Flannel is IP space conservation. In corporate environments where the network team has allocated a very small VCN CIDR block (such as a /24 allowing only 256 addresses), deploying hundreds of pods with OCI-VCN CNI would result in rapid IP exhaustion. Flannel consumes only one primary private VCN IP address per worker node, regardless of whether that node hosts 10 or 100 pods.

Operational Limitations of Flannel

• Encapsulation Overhead: The VXLAN header adds about 50 bytes to every packet, so the pod interface MTU must be 50 bytes smaller than the node's MTU (for example, 8,950 bytes on a 9,000-byte VCN). Encapsulation and decapsulation also consume worker CPU.
• Non-Routable Pod IPs: Flannel pod IPs are completely invisible and unroutable outside the Kubernetes cluster. On-premises systems or external VCN instances cannot route directly to a pod.
• Egress SNAT: When a pod initiates a connection to an external resource (such as an Autonomous Database or on-premises server), the worker node applies Source Network Address Translation (SNAT), replacing the pod IP with the node's primary private IP. Consequently, target systems and VCN Flow Logs see only the host node IP, eliminating pod-level attribution for security auditing.

---

Subnet CIDR Sizing & Capacity Planning

When implementing OCI-VCN CNI, inadequate subnet sizing can halt cluster scaling. Architects must calculate required address space for both the Node Subnet and the Pod Subnet.

Sizing Formula for OCI-VCN CNI Pod Subnets

The required pod subnet CIDR capacity is governed by the maximum anticipated cluster scale: Required Pod Subnet IPs = (Max Worker Nodes * Max Pods Per Node) + OCI Reserved IPs

The maximum pods per node depends on the shape's VNIC limit: MIN((number of VNICs − 1) × 31, 256). Lowering the configured maximum pods per node reduces the number of IP addresses pre-allocated in the pod subnet.

OCI permanently reserves 3 IP addresses in every subnet (the network address, default gateway, and broadcast address).

Sizing Example

Consider an enterprise cluster designed to scale to a maximum of 100 worker nodes, with each node's shape supporting two VNICs, which allows up to 31 pods per node ((2 − 1) × 31):

• Active pod capacity: 100 nodes multiplied by 31 pods = 3,100 IP addresses.
• Rolling node replacement buffer: During a Kubernetes version upgrade, new nodes are provisioned before old nodes are terminated. A 20% surge requires 20 additional nodes: 20 nodes multiplied by 31 pods = 620 additional pod IPs.
• Total required IPs: 3,100 + 620 + 3 = 3,723 IP addresses.
• Subnet selection: A /21 subnet provides 2,048 addresses (insufficient). The architect must provision a /20 subnet, which provides 4,096 IP addresses, ensuring adequate headroom for scaling and non-disruptive upgrades.

---

Ingress & Service Load Balancing: OCI NLB vs. Flexible ALB

Exposing microservices to external consumers or internal VCN clients is managed by the OKE OCI Cloud Controller Manager (CCM) through Kubernetes type: Service LoadBalancer definitions. Architects control load balancer provisioning using declarative annotations:

1. OCI Network Load Balancer (NLB) — Layer 4

The OCI Network Load Balancer provides non-terminating, pass-through Layer 4 (TCP/UDP) routing engineered for extreme performance:

• Throughput & Latency: Designed for high-throughput, low-latency Layer 4 load balancing of TCP and UDP traffic.
• Source IP Preservation: Natively preserves the client's original source IP address without requiring HTTP X-Forwarded-For headers or Proxy Protocol injection.
• Annotation: Enabled by specifying oci.oraclecloud.com/load-balancer-type: "nlb" in the Service manifest.

2. OCI Flexible Load Balancer (ALB) — Layer 7

The OCI Flexible Load Balancer provides comprehensive Layer 7 HTTP/HTTPS application routing:

• Advanced Traffic Routing: Supports SSL/TLS termination, HTTP path-based and host-based routing, cookie-based session persistence, and integration with OCI Web Application Firewall (WAF).
• Dynamic Bandwidth: Configured with dynamic minimum and maximum bandwidth limits (e.g., min 10 Mbps, max 400 Mbps) to scale automatically with incoming request volume.
• Annotations: Enabled with service.beta.kubernetes.io/oci-load-balancer-shape: "flexible" along with flex-min and flex-max bandwidth specifications.

---

Networking Architectural Tradeoff Matrix