11.5 Application Debugging & Developer Workflows in Kubernetes

11.5 Application Debugging & Developer Workflows in Kubernetes

Quick Answer: The official KCNA curriculum lists Debugging as a competency under Cloud Native Application Delivery. The core tools are kubectl debug (attach an ephemeral container to a running Pod, or clone the Pod with a changed image or command), kubectl port-forward (tunnel a local port to an in-cluster endpoint), kubectl cp (move files in and out), and kubectl debug node/<name> (a privileged Pod on the host). Around them sits the inner-loop tooling — Skaffold, Tilt, Telepresence, DevSpace — that makes day-to-day development bearable.

Section 10.1 covered diagnosing failing containers. This section covers working on an application that is already running.

---

1. The Distroless Problem

Section 6.1 recommended minimal base images: distroless or scratch, no shell, no package manager, no curl. That is excellent security and terrible ergonomics — kubectl exec -it pod -- /bin/sh fails with executable file not found because there is genuinely no shell in the image.

Ephemeral containers resolve the conflict. kubectl debug adds a temporary container to an already-running Pod, sharing its network namespace and — with --target — its process namespace:

kubectl debug -it payments-7d9f8-x2k4 \
  --image=busybox:1.36 \
  --target=api \
  -- sh

Inside that ephemeral container you can now:

• wget -qO- http://localhost:8080/healthz — the app's own port, because the network namespace is shared,
• nslookup payments.finance.svc.cluster.local — test DNS from exactly this Pod's perspective,
• ps aux — see the application's processes, because --target shares the process namespace,
• inspect /proc/1/environ and mounted volumes.

Properties worth memorising: an ephemeral container cannot be added to a Pod template, has no resource requests or limits, cannot have probes, and is never restarted. It exists to debug one Pod, once.

---

2. Debugging a Pod That Will Not Start

Ephemeral containers only help a Pod that is running. For a Pod stuck in CrashLoopBackOff, kubectl debug can instead build a copy:

# Copy the Pod but replace the entrypoint with a shell,
# so the failing command never runs and you can inspect the filesystem
kubectl debug payments-7d9f8-x2k4 -it \
  --copy-to=payments-debug \
  --container=api \
  -- sh

# Copy the Pod but swap in a fatter image that has debugging tools
kubectl debug payments-7d9f8-x2k4 \
  --copy-to=payments-debug \
  --set-image=api=myapp:v2.1.0-debug

The copy keeps the original's volumes, environment, and ServiceAccount, so configuration and mount problems reproduce faithfully — while the original Pod is left untouched for the record. Delete the copy when finished.

---

3. Reaching Into the Cluster

kubectl port-forward

Tunnels a local port to a Pod or Service through the API server:

kubectl port-forward pod/payments-7d9f8 8080:8080
kubectl port-forward svc/postgres 5432:5432 -n database
kubectl port-forward deploy/grafana 3000:3000 -n monitoring

This is how you open a database GUI against an in-cluster database, or reach an internal admin endpoint, without creating a NodePort or a LoadBalancer. Traffic rides the authenticated API connection, so no ingress path is exposed and your existing RBAC still applies. Note that for a Service it forwards to one backing Pod, not a load-balanced set.

kubectl proxy

Runs an authenticated local proxy to the API itself, useful for exploring raw API endpoints and reaching Service URLs through the API server:

kubectl proxy --port=8001
curl http://localhost:8001/api/v1/namespaces/production/pods

kubectl cp

kubectl cp production/payments-7d9f8:/app/heapdump.hprof ./heapdump.hprof
kubectl cp ./fixture.json production/payments-7d9f8:/tmp/fixture.json

Pull a heap dump, core file, or log out for offline analysis; push a test fixture in. (tar must exist in the container, so distroless images need the ephemeral-container route instead.)

---

4. Debugging the Node

When the problem is beneath the Pod, kubectl debug can open a privileged Pod in the host's namespaces, with the host filesystem mounted at /host:

kubectl debug node/worker-07 -it --image=ubuntu
# then, inside:
chroot /host
journalctl -u kubelet -n 100
crictl ps -a
df -h

This is the supported alternative to SSH-ing into nodes, and it is exactly the capability that Pod Security Standards restrict — which is why it should be gated by RBAC and audited.

---

5. Reading Application Failures

A disciplined bottom-up sequence — DNS resolves, TCP connects, HTTP responds — isolates the layer in three commands instead of an afternoon.

---

6. The Inner Loop

The naive Kubernetes development cycle is edit → build image → push → update manifest → wait for rollout → test, which can take minutes per keystroke-level change. Inner-loop tools compress it:

Telepresence deserves the emphasis: it lets a developer set an ordinary breakpoint in their IDE and have real production-shaped traffic hit it, which no amount of log-reading matches.

---

7. Debugging Etiquette in Shared Clusters

• Prefer --copy-to over debugging the live Pod when the workload is serving traffic.
• Delete debug copies and ephemeral workloads afterwards — they consume quota and confuse the next person.
• Never leave a port-forward running to a production database.
• Remember that ephemeral containers and node debug Pods are audited privileged actions, and treat the access accordingly.