6.11 NetworkPolicy Isolation & CoreDNS Resolution Diagnostics
Key Takeaways
- Pods are non-isolated by default. Once selected for ingress or egress isolation, traffic in that direction is allowed by the union of applicable NetworkPolicy rules and otherwise denied.
- NetworkPolicy enforcement requires a CNI plugin with policy engine support (such as Calico, Cilium, or Antrea); standard Flannel does NOT enforce NetworkPolicies.
- A critical NetworkPolicy pitfall is blocking DNS: an isolated Pod needs UDP and TCP port 53 egress to the cluster DNS endpoints, typically CoreDNS Pods selected in kube-system.
- In NetworkPolicy YAML, array items in the 'from' or 'to' blocks evaluate with OR logic, whereas multiple selectors inside a single object evaluate with AND logic.
- CoreDNS issues (crash loops, lookup timeouts) typically stem from upstream DNS forward loops detected by the CoreDNS 'loop' plugin, misconfigured Corefile ConfigMaps, or network policy blocking.
NetworkPolicy Isolation & CoreDNS Resolution Diagnostics
Network security in Kubernetes relies on NetworkPolicies—declarative layer 3 and layer 4 packet filtering rules evaluated and enforced by the cluster's Container Network Interface (CNI) plugin. When network policies are misconfigured, applications experience silent connection timeouts, broken database links, and failed health checks.
Similarly, CoreDNS provides the internal service discovery backbone for the entire cluster. If DNS resolution breaks or degrades, microservices cannot locate peer services by name, leading to cascading application outages. Mastering NetworkPolicy isolation diagnostics and CoreDNS debugging is critical for both production cluster reliability and the CKA exam.
1. NetworkPolicy Mechanics & Default-Allow to Default-Deny Boundaries
By default, Kubernetes network communication is non-isolated (default-allow): every Pod can communicate with every other Pod across all namespaces and worker nodes.
+-----------------------------------------------------------------------------------------+
| NETWORKPOLICY ISOLATION TRANSITION |
| |
| [DEFAULT STATE: NON-ISOLATED] |
| - All Ingress traffic allowed from any source. |
| - All Egress traffic allowed to any destination. |
| | |
| v (NetworkPolicy created selecting Pod) |
| [ISOLATED STATE: DEFAULT-DENY] |
| - If policyTypes includes 'Ingress': ALL incoming traffic is DENIED except whitelisted.|
| - If policyTypes includes 'Egress': ALL outgoing traffic is DENIED except whitelisted. |
+-----------------------------------------------------------------------------------------+
[!IMPORTANT] CNI Plugin Policy Support Requirement: NetworkPolicies are API specifications enforced exclusively by the CNI plugin. If a cluster runs standard Flannel without network policy extensions, NetworkPolicy objects can be created in the API server, but no network traffic will ever be filtered or blocked. A CNI with native policy enforcement (e.g., Calico, Cilium, Antrea, Weave Net, or Canal) is mandatory.
2. The CoreDNS Egress Trap & Whitelisting Rule
The most frequent administrative failure when implementing an Egress NetworkPolicy is failing to whitelist internal DNS resolution on UDP/TCP port 53. If you isolate egress traffic without permitting DNS queries, the Pod cannot resolve any service name (including kubernetes.default), causing all outbound network calls to time out.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: secure-app-policy
namespace: production
spec:
podSelector:
matchLabels:
app: api-gateway
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector:
matchLabels:
role: frontend
ports:
- protocol: TCP
port: 8080
egress:
# Allow DNS to CoreDNS Pods in kube-system.
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
# --- ALLOW OUTBOUND TO DATABASE ---
- to:
- podSelector:
matchLabels:
app: postgres-db
ports:
- protocol: TCP
port: 5432
3. AND vs. OR Logic in NetworkPolicy Rules
Misaligning list dashes (-) in NetworkPolicy YAML fundamentally alters evaluation logic:
+-----------------------------------------------------------------------------------------+
| NETWORKPOLICY SELECTOR LOGIC EVALUATION |
| |
| [EXAMPLE A: OR LOGIC (Two Separate List Items)] |
| ingress:
| - from:
| - namespaceSelector: # Match ANY pod in namespace with label env=prod |
| matchLabels: # OR |
| env: prod # Match pods in LOCAL namespace with role=client |
| - podSelector: |
| matchLabels: |
| role: client |
| |
| [EXAMPLE B: AND LOGIC (Single List Item with Multiple Selectors)] |
| ingress:
| - from:
| - namespaceSelector: # Match ONLY pods with label role=client |
| matchLabels: # INSIDE namespaces with label env=prod |
| env: prod # (BOTH conditions must be satisfied) |
| podSelector: |
| matchLabels: |
| role: client |
+-----------------------------------------------------------------------------------------+
Combining ipBlock, namespaceSelector, and podSelector
ipBlock: Specifies CIDR ranges (e.g.,cidr: 10.0.0.0/16,except: [10.0.1.0/24]). Cannot be used to target cluster Pod IPs directly because Pod IPs are dynamic.namespaceSelector: Selects entire namespaces based on namespace labels (e.g.,kubernetes.io/metadata.name: kube-system).podSelector: Selects Pods based on Pod labels within the target namespace.
4. CoreDNS Architecture & Diagnostic Runbook
CoreDNS runs as a 2-replica Deployment in the kube-system namespace, exposed via a ClusterIP service (usually 10.96.0.10 in kubeadm clusters).
+-----------------------------------------------------------------------------------------+
| DNS RESOLUTION & /etc/resolv.conf |
| |
| [INSIDE POD: /etc/resolv.conf] |
| nameserver 10.96.0.10 |
| search default.svc.cluster.local svc.cluster.local cluster.local |
| options ndots:5 |
| |
| QUERY RESOLUTION ORDER for 'backend-svc': |
| 1. backend-svc.default.svc.cluster.local ---> SUCCESS (Returns ClusterIP) |
| 2. backend-svc.svc.cluster.local ---> NXDOMAIN |
| 3. backend-svc.cluster.local ---> NXDOMAIN |
+-----------------------------------------------------------------------------------------+
The ndots:5 Resolution Penalty
By default, Kubernetes configures /etc/resolv.conf with options ndots:5. Any query with fewer than 5 dots (e.g., api.example.com has 2 dots) will first append the search domains (default.svc.cluster.local, svc.cluster.local, cluster.local) before querying the public root servers, generating 3 NXDOMAIN queries for every external lookup. In high-traffic workloads, using Fully Qualified Domain Names ending with a dot (e.g., api.example.com.) bypasses search domain expansion.
Diagnosing CoreDNS Outages:
# 1. Check CoreDNS deployment and pod health
kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide
# 2. Check CoreDNS logs for errors, forward timeouts, or crash loops
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50
# 3. Check CoreDNS Corefile ConfigMap
kubectl get configmap coredns -n kube-system -o yaml
# 4. Enable CoreDNS query logging for real-time inspection
# Edit ConfigMap and add the 'log' plugin inside the Corefile block
kubectl edit configmap coredns -n kube-system
The CoreDNS loop CrashLoopBackOff:
If CoreDNS pods are in CrashLoopBackOff with the log:
plugin/loop: Loop (127.0.0.1:53 -> :53) detected for zone "."
- Cause: Upstream DNS forward loop caused by
systemd-resolvedon the host node forwarding queries back to127.0.0.53. - Remediation: Keep the protective
loopplugin. On systemd-resolved hosts, configure kubeletresolvConf(or--resolv-conf) to use the real resolver file, commonly/run/systemd/resolve/resolv.conf; kubeadm normally detects this. Alternatively configure CoreDNSforwardto an approved non-looping upstream. Then restart CoreDNS and verify queries.
5. End-to-End DNS Verification Commands
To verify DNS resolution from within a live workload:
# Launch an ephemeral diagnostic container
kubectl run dns-test --rm -it --image=busybox:1.36 --restart=Never -- /bin/sh
# --- Inside test container ---
# Test cluster-internal service resolution
nslookup kubernetes.default
nslookup backend-svc.production.svc.cluster.local
# Test direct query against CoreDNS Service IP
nslookup kubernetes.default 10.96.0.10
# Test SRV record discovery for headless StatefulSet services
nslookup -type=SRV cassandra-headless.default.svc.cluster.local
# Test external Internet resolution
nslookup google.com
An administrator applies an Egress NetworkPolicy to isolate a secure payment pod in the finance namespace. Immediately after applying the policy, the payment pod is unable to connect to any other service by hostname, and all outbound requests time out. What essential egress rule was omitted from the NetworkPolicy specification?
A cluster uses pure Flannel as its Container Network Interface (CNI). A security engineer creates a NetworkPolicy designed to deny all ingress traffic to namespace secure-data. However, pods in other namespaces can still freely establish connections to pods in secure-data. What is the cause of this behavior?
Both CoreDNS pods in the kube-system namespace are in CrashLoopBackOff. Inspecting the logs with kubectl logs -n kube-system -l k8s-app=kube-dns reveals: plugin/loop: Loop (127.0.0.1:53 -> :53) detected for zone ".". What is the root cause of this failure?
You've completed this section
Continue exploring other exams