7.2 Cluster DNS & Service Discovery with CoreDNS

7.2 Cluster DNS & Service Discovery with CoreDNS

Quick Answer: Kubernetes ships an in-cluster DNS server — CoreDNS, a CNCF graduated project — running as a Deployment in kube-system and fronted by a Service conventionally named kube-dns. Every Service receives an A/AAAA record at <service>.<namespace>.svc.cluster.local pointing at its ClusterIP. The kubelet writes a search list into every Pod's /etc/resolv.conf, which is why curl http://payments works inside the same namespace but a cross-namespace call needs payments.finance.

Section 7.1 explained that a Service provides a stable virtual IP. This section explains how a Pod ever learns that IP in the first place — and why "it's always DNS" is the oldest joke in platform engineering.

---

1. Why Discovery Must Be Dynamic

Pod IPs change on every reschedule. Hard-coding one is meaningless. Kubernetes therefore layers two stable abstractions on top:

Pod  ──asks──►  CoreDNS  ──returns──►  Service ClusterIP
                                            │
                                       kube-proxy rules
                                            ▼
                             one of the healthy backing Pod IPs

The application only ever needs to know a name. CoreDNS turns the name into a ClusterIP; kube-proxy turns the ClusterIP into a live endpoint.

---

2. CoreDNS

CoreDNS replaced the older kube-dns add-on as the cluster DNS default in Kubernetes 1.13. It is a single Go binary built from chained plugins, configured through a Corefile stored in a ConfigMap:

.:53 {
    errors
    health
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
       pods insecure
       fallthrough in-addr.arpa ip6.arpa
    }
    prometheus :9153     # exposes CoreDNS metrics for scraping
    forward . /etc/resolv.conf     # anything not cluster.local goes upstream
    cache 30
    loop
    reload
    loadbalance
}

Key facts KCNA expects:

• CoreDNS runs as a Deployment (typically two replicas) in kube-system, not a DaemonSet.
• It watches the Kubernetes API for Services and EndpointSlices and answers from that live view — it is not a zone file that needs reloading.
• The forward plugin sends non-cluster names (github.com) to the node's upstream resolvers.
• The prometheus plugin makes CoreDNS observable, which matters because DNS latency shows up as application latency everywhere at once.
• NodeLocal DNSCache is an optional DaemonSet that puts a caching resolver on every node, cutting cross-node DNS traffic and conntrack pressure on large clusters.

---

3. The DNS Name Format

Services

<service-name>.<namespace>.svc.cluster.local

The Search List

The kubelet writes something like this into each Pod:

nameserver 10.96.0.10
search production.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

So, from a Pod in namespace production:

The ndots:5 gotcha. Any name containing fewer than five dots is tried against every search-domain suffix first. Looking up api.github.com (two dots) therefore generates three failing cluster queries before the correct one. On busy clusters this is a measurable source of DNS load and tail latency; the fix is a trailing dot (api.github.com.) or a per-Pod dnsConfig lowering ndots.

Pods

Pods also get records, with dots in the IP replaced by dashes: 10-244-1-15.production.pod.cluster.local. These are rarely used directly — except by StatefulSets.

---

4. Headless Services and StatefulSet Identity

Setting clusterIP: None creates a headless Service: no virtual IP is allocated and kube-proxy programs no rules. CoreDNS instead returns the A records of all ready backing Pods.

apiVersion: v1
kind: Service
metadata:
  name: redis-service
spec:
  clusterIP: None          # headless
  selector:
    app: redis
  ports:
  - port: 6379

Combined with a StatefulSet's serviceName, this produces the stable per-Pod names that clustered databases require:

redis-cluster-0.redis-service.production.svc.cluster.local
redis-cluster-1.redis-service.production.svc.cluster.local
redis-cluster-2.redis-service.production.svc.cluster.local

A Cassandra or Kafka node needs to address a specific peer, not "whichever replica the load balancer picks" — headless DNS is what makes that possible.

---

5. DNS Policies

spec.dnsPolicy controls what the kubelet writes into resolv.conf:

Trap: a Pod with hostNetwork: true and the default ClusterFirst silently gets the node's resolver and cannot resolve Service names. ClusterFirstWithHostNet is the fix.

---

6. Environment-Variable Discovery

Kubernetes also injects Docker-link-style variables for every Service that existed before the Pod started:

PAYMENTS_SERVICE_HOST=10.96.31.7
PAYMENTS_SERVICE_PORT=80

This mechanism is legacy and has a fatal ordering flaw: a Service created after the Pod produces no variables, so it silently fails to the application. DNS is the correct answer for service discovery in any modern cluster; environment variables exist for backwards compatibility only.

---

7. Triaging DNS Failures

kubectl get pods -n kube-system -l k8s-app=kube-dns     # are CoreDNS Pods Ready?
kubectl logs -n kube-system -l k8s-app=kube-dns          # SERVFAIL? loop detected?
kubectl run tmp --rm -it --image=busybox:1.36 --restart=Never -- \
  nslookup payments.finance.svc.cluster.local            # resolve from inside
kubectl exec <pod> -- cat /etc/resolv.conf               # right nameserver and search list?
kubectl get endpointslices -l kubernetes.io/service-name=payments