Troubleshooting¶

Reliable Kubernetes troubleshooting is a workflow, not a command list.

Start with symptoms, follow evidence, and narrow the failure domain quickly.

Triage sequence¶

flowchart TD
    START([Pod not working]) --> STATUS{kubectl get pod\nstatus?}
    STATUS -->|Pending| PEND[Check node resources\ntaints and tolerations\ncheck describe pod]
    STATUS -->|ImagePullBackOff| IMG[Check image name/tag\npull secret\nregistry auth]
    STATUS -->|CrashLoopBackOff| CRASH[Check logs --previous\ncheck exit code\ncheck liveness probe]
    STATUS -->|Running but broken| CONN[Check readiness probe\ncheck service selector\ncheck endpoint slices]
    PEND --> FIX([Fix and redeploy])
    IMG --> FIX
    CRASH --> FIX
    CONN --> FIX

Detailed sequence:

1. Identify failing object type and status.
2. Inspect events and controller messages (kubectl describe).
3. Inspect application and sidecar logs (kubectl logs --previous for crashed containers).
4. Validate config references and runtime environment (secrets, configmaps, volumes).
5. Test service connectivity and DNS paths.

First-response commands¶

kubectl get pods -A
kubectl describe pod <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace> --all-containers
kubectl logs <pod-name> -n <namespace> --all-containers --previous
kubectl get events -A --sort-by=.metadata.creationTimestamp

Common status patterns¶

CrashLoopBackOff uses exponential backoff: 10s, 20s, 40s, 80s, 160s, then 5 minutes (capped). This means a crashing pod can be slow to recover even after you fix the root cause.

Exit codes narrow the cause fast:

For OOM kills specifically, kubectl describe pod shows Last State: Terminated, Reason: OOMKilled. Note that the OOM killer targets a process, and a pod can be OOM-killed even when the node has free memory -- the container's own limit is the ceiling that matters.

Reading Pending correctly¶

Pending means the scheduler cannot place the pod, and kubectl describe pod always tells you why in the events -- read the FailedScheduling message closely. The format 0/12 nodes are available: 8 Insufficient cpu, 4 node(s) had untolerated taint {...} is an exhaustive census: every node appears in exactly one bucket. Typical causes, in rough order of frequency:

1. Insufficient resources: no node has enough unrequested (not unused) CPU or memory. Fix requests, add nodes, or wait for the autoscaler.
2. Untolerated taints: including the implicit ones on control-plane nodes and NotReady nodes.
3. Affinity/selector constraints: nodeSelector labels that match nothing, or anti-affinity that conflicts with existing pods.
4. Volume topology: the PVC's volume lives in a zone where no eligible node exists.

For the full placement model behind these messages, see Scheduling and Placement.

Network diagnosis flow¶

kubectl get svc -n <namespace>
kubectl get endpointslices -n <namespace>
kubectl exec -it <pod-name> -n <namespace> -- nslookup <service>
kubectl exec -it <pod-name> -n <namespace> -- wget -qO- http://<service>:<port>

If service has no endpoints, verify selector labels and readiness state of backend pods.

Debugging running containers¶

Use kubectl exec for interactive inspection. For minimal images or crash loops, use ephemeral debug containers:

kubectl debug -it <pod-name> -n <namespace> --image=busybox:1.36 --target=<container-name>

Control plane and node checks¶

kubectl get nodes
kubectl describe node <node-name>
kubectl top nodes
kubectl top pods -A

For node pressure or kubelet issues, inspect node conditions and recent events.

Incident habits that reduce MTTR¶

• document exact failing timestamp and first observed symptom
• capture commands and outputs in a runbook timeline
• avoid changing multiple variables at once during diagnosis
• confirm recovery with objective service checks

Certification notes¶

• Troubleshooting is the heaviest-weighted CKA domain (30%). The triage sequence above -- status, describe/events, logs, config, connectivity -- is exactly the order that solves exam scenarios fastest.
• For "node NotReady" tasks: ssh to the node, systemctl status kubelet, journalctl -u kubelet -- the fix is usually starting/enabling the kubelet.
• Learn the exit-code table above; it converts a describe output into a diagnosis in seconds.

Summary¶

Effective Kubernetes troubleshooting depends on sequence and discipline. Start with status and events, then move to logs, configuration, and connectivity checks in a controlled order.

Related Concepts¶

• Pods and Deployments
• Networking Overview
• Security Primer