11.5 Application Debugging & Developer Workflows in Kubernetes

Key Takeaways

kubectl debug attaches an ephemeral container to a running Pod, which is the only way to get a shell alongside a distroless or scratch-based container.
kubectl debug can also create a copy of a Pod with a modified command or image, useful when the original crashes before a shell can attach.
kubectl port-forward tunnels a local port to a Pod or Service so a developer can reach an internal endpoint without exposing it publicly.
kubectl cp copies files between a local machine and a container, and kubectl debug node/<name> opens a privileged debugging Pod on a node host.
Inner-loop tools such as Skaffold, Tilt, Telepresence, and DevSpace shorten the code-build-deploy cycle that otherwise makes Kubernetes development slow.

Last updated: August 2026

11.5 Application Debugging & Developer Workflows in Kubernetes

Quick Answer: The official KCNA curriculum lists Debugging as a competency under Cloud Native Application Delivery. The core tools are kubectl debug (attach an ephemeral container to a running Pod, or clone the Pod with a changed image or command), kubectl port-forward (tunnel a local port to an in-cluster endpoint), kubectl cp (move files in and out), and kubectl debug node/<name> (a privileged Pod on the host). Around them sits the inner-loop tooling — Skaffold, Tilt, Telepresence, DevSpace — that makes day-to-day development bearable.

Section 10.1 covered diagnosing failing containers. This section covers working on an application that is already running.

1. The Distroless Problem

Section 6.1 recommended minimal base images: distroless or scratch, no shell, no package manager, no curl. That is excellent security and terrible ergonomics — kubectl exec -it pod -- /bin/sh fails with executable file not found because there is genuinely no shell in the image.

Ephemeral containers resolve the conflict. kubectl debug adds a temporary container to an already-running Pod, sharing its network namespace and — with --target — its process namespace:

kubectl debug -it payments-7d9f8-x2k4 \
  --image=busybox:1.36 \
  --target=api \
  -- sh

Inside that ephemeral container you can now:

wget -qO- http://localhost:8080/healthz — the app's own port, because the network namespace is shared,
nslookup payments.finance.svc.cluster.local — test DNS from exactly this Pod's perspective,
ps aux — see the application's processes, because --target shares the process namespace,
inspect /proc/1/environ and mounted volumes.

Properties worth memorising: an ephemeral container cannot be added to a Pod template, has no resource requests or limits, cannot have probes, and is never restarted. It exists to debug one Pod, once.

2. Debugging a Pod That Will Not Start

Ephemeral containers only help a Pod that is running. For a Pod stuck in CrashLoopBackOff, kubectl debug can instead build a copy:

# Copy the Pod but replace the entrypoint with a shell,
# so the failing command never runs and you can inspect the filesystem
kubectl debug payments-7d9f8-x2k4 -it \
  --copy-to=payments-debug \
  --container=api \
  -- sh

# Copy the Pod but swap in a fatter image that has debugging tools
kubectl debug payments-7d9f8-x2k4 \
  --copy-to=payments-debug \
  --set-image=api=myapp:v2.1.0-debug

The copy keeps the original's volumes, environment, and ServiceAccount, so configuration and mount problems reproduce faithfully — while the original Pod is left untouched for the record. Delete the copy when finished.

3. Reaching Into the Cluster

`kubectl port-forward`

Tunnels a local port to a Pod or Service through the API server:

kubectl port-forward pod/payments-7d9f8 8080:8080
kubectl port-forward svc/postgres 5432:5432 -n database
kubectl port-forward deploy/grafana 3000:3000 -n monitoring

This is how you open a database GUI against an in-cluster database, or reach an internal admin endpoint, without creating a NodePort or a LoadBalancer. Traffic rides the authenticated API connection, so no ingress path is exposed and your existing RBAC still applies. Note that for a Service it forwards to one backing Pod, not a load-balanced set.

`kubectl proxy`

Runs an authenticated local proxy to the API itself, useful for exploring raw API endpoints and reaching Service URLs through the API server:

kubectl proxy --port=8001
curl http://localhost:8001/api/v1/namespaces/production/pods

`kubectl cp`

kubectl cp production/payments-7d9f8:/app/heapdump.hprof ./heapdump.hprof
kubectl cp ./fixture.json production/payments-7d9f8:/tmp/fixture.json

Pull a heap dump, core file, or log out for offline analysis; push a test fixture in. (tar must exist in the container, so distroless images need the ephemeral-container route instead.)

4. Debugging the Node

When the problem is beneath the Pod, kubectl debug can open a privileged Pod in the host's namespaces, with the host filesystem mounted at /host:

kubectl debug node/worker-07 -it --image=ubuntu
# then, inside:
chroot /host
journalctl -u kubelet -n 100
crictl ps -a
df -h

This is the supported alternative to SSH-ing into nodes, and it is exactly the capability that Pod Security Standards restrict — which is why it should be gated by RBAC and audited.

5. Reading Application Failures

Symptom	Where to look
Container starts then exits 0 immediately	The image's entrypoint completed — a foreground process is required, not a background daemon
`exec format error`	Architecture mismatch: an `amd64` image on an ARM node. Build a multi-arch image (section 6.2)
`permission denied` writing a file	`readOnlyRootFilesystem: true`, or `runAsUser` does not own the path — add an `emptyDir` or set `fsGroup`
Env var empty although a ConfigMap exists	The key name is wrong, or the ConfigMap is in a different namespace
App cannot reach a dependency	Test from an ephemeral container: DNS first, then TCP, then HTTP
Works locally, fails in-cluster	Almost always configuration, DNS, or a NetworkPolicy — compare the two environments explicitly

A disciplined bottom-up sequence — DNS resolves, TCP connects, HTTP responds — isolates the layer in three commands instead of an afternoon.

6. The Inner Loop

The naive Kubernetes development cycle is edit → build image → push → update manifest → wait for rollout → test, which can take minutes per keystroke-level change. Inner-loop tools compress it:

Tool	Approach
Skaffold	Watches source, rebuilds and redeploys automatically, streams logs back; profiles for dev vs prod
Tilt	Similar loop with a live web UI showing every service's build and runtime status; supports live file sync into running containers
Telepresence	Reroutes traffic for one in-cluster Service to a process on your laptop, so you can use a normal local debugger against real cluster dependencies
DevSpace	Dev containers with file sync and hot reload
`kind` / minikube	A disposable local cluster so the whole loop runs offline

Telepresence deserves the emphasis: it lets a developer set an ordinary breakpoint in their IDE and have real production-shaped traffic hit it, which no amount of log-reading matches.

7. Debugging Etiquette in Shared Clusters

Prefer --copy-to over debugging the live Pod when the workload is serving traffic.
Delete debug copies and ephemeral workloads afterwards — they consume quota and confuse the next person.
Never leave a port-forward running to a production database.
Remember that ephemeral containers and node debug Pods are audited privileged actions, and treat the access accordingly.

Test Your Knowledge

A container is built from a distroless base image and kubectl exec -it pod -- /bin/sh fails with executable file not found. What is the correct approach?

Rebuild the image on a full Ubuntu base so a shell is always available in production

Use kubectl debug to attach an ephemeral container with a shell, sharing the Pod's network and optionally its process namespace

Enable privileged mode on the container so exec can inject a shell

Use kubectl cp to copy a shell binary into the running container

Test Your Knowledge

A Pod is in CrashLoopBackOff and exits before any shell can attach. Which kubectl debug form helps?

kubectl debug with --target, attaching an ephemeral container to the crashing Pod

kubectl debug with --copy-to, creating a copy of the Pod whose entrypoint is replaced by a shell so the failing command never runs

kubectl debug node/<name>, opening a privileged Pod on the host

kubectl port-forward to the Pod's application port

Test Your Knowledge

What does kubectl port-forward svc/postgres 5432:5432 accomplish?

It changes the Service type to NodePort so the database is reachable from outside the cluster

It tunnels a local port through the authenticated API connection to one backing Pod, so local tools can reach the in-cluster database without exposing it publicly

It permanently publishes the Service through the Ingress controller

It copies the database files to the local machine

Sections you finish are checked off in the contents.

Up Next

11.6 Platform Engineering & Developer Self-Service

Continue learning