6.9 Service, Endpoint & Ingress Connectivity Troubleshooting

6.9 Service, Endpoint & Ingress Connectivity Troubleshooting

Networking in Kubernetes operates across multiple virtual layers: container virtual ethernet pairs (veth), CNI flat routing, virtual ClusterIP load-balancing via kube-proxy, cluster-internal DNS resolution via CoreDNS, and HTTP/HTTPS layer 7 ingress routing.

When a client or frontend microservice cannot communicate with a backend service, administrators must isolate the exact hop where traffic is dropped.

1. End-to-End Traffic Path & Hop-by-Hop Triage

+-----------------------------------------------------------------------------------------+
|                          END-TO-END SERVICE TRAFFIC PIPELINE                            |
|                                                                                         |
|  [CLIENT POD]                                                                           |
|         |                                                                               |
|         v (1. Resolves DNS: backend-svc.default.svc.cluster.local -> 10.96.0.50)        |
|  [CoreDNS]                                                                              |
|         |                                                                               |
|         v (2. Sends packet to ClusterIP 10.96.0.50:80)                                  |
|  [KUBE-PROXY / IPTABLES / IPVS on Host Kernel]                                          |
|  - Matches DNAT rule for 10.96.0.50:80                                                  |
|  - Randomly selects backend Pod IP from Endpoints / EndpointSlices (e.g., 10.244.1.25)  |
|  - Rewrites destination IP to 10.244.1.25:8080 (targetPort)                             |
|         |                                                                               |
|         v (3. CNI Routes packet across node network overlay)                            |
|  [TARGET BACKEND POD (10.244.1.25)]                                                     |
|  - Listens on container port 8080                                                       |
|  - Readiness probe must be PASSING (otherwise omitted from Endpoints)                   |
+-----------------------------------------------------------------------------------------+

2. Service & Endpoints Diagnostic Checklist

When a ClusterIP or NodePort Service fails to route traffic, execute this sequential 5-step checklist:

# Step 1: Verify the Service exists and has an assigned ClusterIP
kubectl get svc backend-svc

# Step 2: Check if Endpoints and EndpointSlices exist for the service
kubectl get endpoints backend-svc
kubectl get endpointslices -l kubernetes.io/service-name=backend-svc

Case A: Endpoints displays <none>

If kubectl get endpoints backend-svc shows ENDPOINTS: <none>, traffic cannot be routed because the Service has zero healthy backend pods.

[TROUBLESHOOTING ZERO ENDPOINTS]
  1. Check Service Selector: $ kubectl get svc backend-svc -o jsonpath='{.spec.selector}'
     Output: map[app:web-backend env:prod]
  2. Check Pod Labels:      $ kubectl get pods --show-labels
     Output: pod-1: app=web-backend, env=production   <-- MISMATCH! (prod vs production)
  3. Check Pod Health:      $ kubectl get pods -l app=web-backend
     - If Pods are Running but 0/1 Ready -> Readiness probe is failing!

[!IMPORTANT] A pod is added to an Endpoints object only if:

1. All key-value pairs in service.spec.selector match the pod's metadata.labels.
2. The pod is in the Running phase.
3. The pod's readinessProbe is succeeding (or no readiness probe is configured).

Case B: Endpoints Populated, But Connection Refused / Times Out

If endpoints exist (e.g., 10.244.1.25:8080), but curling the ClusterIP fails:

# Verify port mapping configuration in Service spec
kubectl get svc backend-svc -o yaml

spec:
  ports:
  - name: http
    port: 80         # Port exposed on the Service ClusterIP
    targetPort: 8080 # Port where application container actually listens
    protocol: TCP

• TargetPort Mismatch: If the application inside the container listens on port 3000, but targetPort is set to 8080, packets reach the pod but the Linux kernel returns TCP RST (Connection Refused).
• Named Port Typo: If targetPort: http-web is used, ensure the container spec contains ports: [{name: http-web, containerPort: 8080}].

3. Ingress Routing & Controller Diagnostics

An Ingress resource is merely a set of rules; traffic routing requires an active Ingress Controller (e.g., ingress-nginx).

+-----------------------------------------------------------------------------------------+
|                            INGRESS ARCHITECTURAL FLOW                                   |
|                                                                                         |
|  External Client ---> [Ingress Controller (e.g., NGINX Pod / NodePort / LB)]            |
|                             |                                                           |
|                             +---> Evaluates Host: app.example.com                       |
|                             +---> Evaluates Path: /api (Prefix matching)                |
|                             |                                                           |
|                             v (Routes directly to Pod IPs from Endpoints)               |
|                       [Backend Pods]                                                    |
+-----------------------------------------------------------------------------------------+

High-Frequency Ingress Issues & Solutions:

1. Missing or Mismatched ingressClassName:

   spec:
     ingressClassName: nginx   # Must match an existing IngressClass: kubectl get ingressclass
   

2. Path Type Matching (Prefix vs Exact):

   • pathType: Exact matching /api will NOT match /api/v1/users.
   • Use pathType: Prefix with path: /api.

3. Rewrite Annotation Omission: If incoming traffic hits app.example.com/api/users, but the backend application expects requests at /users, include the NGINX rewrite annotation:

   metadata:
     annotations:
       nginx.ingress.kubernetes.io/rewrite-target: /$2
   spec:
     rules:
     - host: app.example.com
       http:
         paths:
         - path: /api(/|$)(.*)
           pathType: ImplementationSpecific
           backend:
             service:
               name: backend-svc
               port:
                 number: 80
   

4. Inspect Ingress Controller Logs:

   kubectl logs -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx --tail=100
   

4. In-Cluster Connectivity Debugging Runbook

To prove whether a failure is DNS-related, Service-related, or container-related, run a temporary interactive test container:

# 1. Launch a temporary debugging pod
kubectl run tmp-net --rm -it --image=busybox:1.36 --restart=Never -- /bin/sh

# --- Inside the test container shell ---

# 2. Test DNS resolution of the service name
nslookup backend-svc
nslookup backend-svc.default.svc.cluster.local

# 3. Test TCP port reachability on ClusterIP
nc -zv 10.96.0.50 80

# 4. Test HTTP payload response
wget -qO- http://backend-svc:80/healthz

# 5. Test direct Pod IP connectivity (Bypassing Service layer)
# Retrieve Pod IP: kubectl get pods -l app=backend -o wide
wget -qO- http://10.244.1.25:8080/healthz