4.4 CoreDNS Configuration, Resolution & Custom Forwarding

4.4 CoreDNS Configuration, Resolution & Custom Forwarding

Service discovery in Kubernetes is powered by CoreDNS, a flexible, extensible DNS server compiled with plugins. Deployed as a standard Deployment in the kube-system namespace, CoreDNS watches the Kubernetes API server for Service and EndpointSlice changes, dynamically serving DNS records matching the canonical Kubernetes DNS specification.

1. Kubernetes DNS Record Specifications

Every object in a Kubernetes cluster is assigned a deterministic Fully Qualified Domain Name (FQDN) adhering to the standard schema:

+-----------------------------------------------------------------------------------------+
|                         KUBERNETES DNS RECORD CONVENTIONS                               |
|                                                                                         |
|   1. STANDARD SERVICE (ClusterIP):                                                      |
|      Format:  <service>.<namespace>.svc.<cluster-domain>                                |
|      Example: payment-svc.production.svc.cluster.local  ---> 10.96.50.120               |
|      * Within same namespace:  'payment-svc'                                           |
|      * Across namespaces:      'payment-svc.production'                                 |
|                                                                                         |
|   2. HEADLESS SERVICE POD (StatefulSets):                                               |
|      Format:  <pod-name>.<service>.<namespace>.svc.<cluster-domain>                     |
|      Example: cassandra-0.cassandra.default.svc.cluster.local ---> 10.244.1.15         |
|                                                                                         |
|   3. STANDARD POD DIRECT A-RECORD:                                                      |
|      Format:  <ip-with-dashes>.<namespace>.pod.<cluster-domain>                         |
|      Example: 10-244-2-8.default.pod.cluster.local ---> 10.244.2.8                      |
|                                                                                         |
|   4. SRV RECORDS (Named Ports):                                                         |
|      Format:  _<port-name>._<proto>.<service>.<namespace>.svc.<cluster-domain>          |
|      Example: _http._tcp.web-svc.default.svc.cluster.local ---> Port 80, Target       |
+-----------------------------------------------------------------------------------------+

2. Pod /etc/resolv.conf Mechanics & The ndots:5 Latency Bottleneck

When kubelet provisions a Pod, it injects a customized /etc/resolv.conf file into the container's filesystem:

nameserver 10.96.0.10
search production.svc.cluster.local svc.cluster.local cluster.local corp.internal
options ndots:5

How ndots:5 Operates:

• ndots:5 means: "If a requested domain name contains fewer than 5 dots (periods), treat it as a relative domain and append each entry in the search list sequentially before attempting an absolute query."

+-----------------------------------------------------------------------------------------+
|                        ndots:5 EXTERNAL DOMAIN LOOKUP PIPELINE                          |
|                                                                                         |
|   Application queries: "api.github.com" (Contains 2 dots: < 5)                          |
|                                                                                         |
|   [Query 1] api.github.com.production.svc.cluster.local     ---> NXDOMAIN               |
|   [Query 2] api.github.com.svc.cluster.local                ---> NXDOMAIN               |
|   [Query 3] api.github.com.cluster.local                    ---> NXDOMAIN               |
|   [Query 4] api.github.com.corp.internal                    ---> NXDOMAIN               |
|   [Query 5] api.github.com.                                 ---> SUCCESS (20.207.73.82) |
|                                                                                         |
|   * Result: 4 wasted DNS round-trips for every external API call!                       |
+-----------------------------------------------------------------------------------------+

Mitigating DNS Latency:

1. Use Trailing Dots for External Calls: In application code, query api.github.com. (with a trailing dot) to force immediate absolute resolution.
2. Customize Pod dnsConfig: Tune ndots per Pod:

   spec:
     dnsConfig:
       options:
         - name: ndots
           value: "2"
   

3. Deploy NodeLocal DNSCache: Runs a lightweight DNS cache daemon on every node (169.254.20.10), eliminating network latency to CoreDNS.

3. Corefile Anatomy & Custom Domain Forwarding

CoreDNS configuration is stored in a ConfigMap named coredns in the kube-system namespace. The configuration uses a structured DSL called the Corefile.

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf {
           max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }
    # Custom Private Enterprise Domain Block
    corp.internal:53 {
        errors
        cache 60
        forward . 192.168.10.50 192.168.10.51
    }

Key CoreDNS Plugins Explained:

• errors: Logs DNS lookup errors to standard output.
• health: Exposes an HTTP health check endpoint on :8080/health.
• ready: Exposes readiness endpoint on :8181/ready for readiness probes.
• kubernetes: The core plugin that answers cluster queries matching cluster.local. pods insecure allows resolving direct Pod IPs without verifying identity.
• forward: Forwards queries that do not match the local zone to upstream resolvers (such as host /etc/resolv.conf or external DNS like 8.8.8.8).
• cache: Caches DNS responses in memory (TTL 30 seconds).
• loop: Detects forwarding loops (e.g., CoreDNS forwarding to itself) and halts the process to prevent CPU starvation.
• reload: Automatically reloads the Corefile within seconds of the ConfigMap being edited without restarting pods.

4. DNS Troubleshooting & Diagnostic Runbook

When diagnosing DNS resolution failures in the CKA exam, execute this sequence:

# 1. Verify CoreDNS pods are running in kube-system
kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide

# 2. Inspect CoreDNS service ClusterIP (must match Pod resolv.conf nameserver)
kubectl get svc -n kube-system kube-dns

# 3. Check CoreDNS logs for errors or forwarding loops
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=100

# 4. Launch an ephemeral debug pod with dnsutils to run queries
kubectl run dnsutils --image=registry.k8s.io/e2e-test-images/jessie-dnsutils:1.3 --restart=Never -i --rm -- nslookup kubernetes.default

# 5. Test specific FQDN and reverse PTR lookups
kubectl run dnsutils --image=registry.k8s.io/e2e-test-images/jessie-dnsutils:1.3 --restart=Never -i --rm -- dig @10.96.0.10 payment-svc.prod.svc.cluster.local

[!IMPORTANT] CoreDNS CrashLoopBackOff via Loop Plugin: If CoreDNS pods enter CrashLoopBackOff with the error Loop (127.0.0.1:53 -> :53) detected for zone ".", it indicates that the host node's /etc/resolv.conf points to 127.0.0.53 (systemd-resolved), which in turn forwards back to CoreDNS. Keep the loop plugin. Point kubelet resolvConf to a real resolver file (commonly /run/systemd/resolve/resolv.conf with systemd-resolved), or configure CoreDNS forward to an approved non-looping upstream. Then restart CoreDNS and verify internal and external queries.