Skip to content

Kubelet and Container Runtime

The control plane decides what should run. The kubelet is where deciding ends and running begins -- it is the node agent that turns a Pod object into actual Linux processes, keeps them healthy, and reports back. This page follows that path all the way down to the kernel.

The kubelet's job, precisely

The kubelet is one more watch-driven controller, with a narrow scope: pods bound to my node. Its reconcile loop:

  1. Watch the API server for pods with spec.nodeName == me (plus static pod manifests on disk).
  2. Compare desired pods against what the container runtime is actually running.
  3. Converge: create sandboxes, pull images, start/stop containers, run probes.
  4. Report truth back: pod status, container states, restart counts, node conditions.

Equally important is what the kubelet does not do: it never decides placement (the scheduler's job), never replaces pods that belong elsewhere (controllers' job), and does not implement Services (kube-proxy's job). One process per node, one responsibility: make my node's assigned pods real and describe them honestly.

From Pod object to Linux processes

The kubelet doesn't run containers itself. It delegates through a layered chain, each layer with a single job:

flowchart LR
    K[kubelet] -->|CRI gRPC\nunix socket| C[containerd\nor CRI-O]
    C -->|starts| S[shim\none per container]
    S -->|invokes| R[runc\nOCI runtime]
    R -->|clone + cgroups\n+ namespaces| P[container process]
    R -.exits immediately.-> S
  • CRI (Container Runtime Interface): a gRPC API the kubelet speaks over a local socket. Because it's an interface, containerd and CRI-O are interchangeable -- and this is why Docker-as-runtime could be removed in 1.24 without changing what images run: Docker images were always OCI images.
  • containerd / CRI-O: manage images, snapshots/layers, and container lifecycle.
  • shim: a tiny per-container process that holds the container's stdio and exit status. Its existence means containerd (and the kubelet!) can restart without killing your containers.
  • runc: sets up namespaces and cgroups, starts the process, then exits. There is no runtime "wrapping" your container at runtime -- your application is a plain kernel process, findable with ps on the node.

The isolation itself is just two kernel features: namespaces decide what the process can see (its own network stack, PIDs, mounts, hostname), and cgroups decide what it can use (the CPU and memory enforcement described in Requests and Limits).

The pause container: the pod made flesh

Run ps on any node and you'll find one pause process per pod. It is the answer to "what is a pod, concretely?"

When the kubelet creates a pod, it first asks the runtime for a sandbox: the pause container, which does nothing but hold the pod's shared namespaces (network, IPC) and sleep. Every app container then joins the pause container's namespaces.

This tiny trick delivers the pod contract:

  • All containers share one IP and can talk over localhost -- they share the pause container's network namespace.
  • An app container can crash and restart without the pod's IP changing -- the namespace lives in pause, not in the app container.
  • The pod "exists" independently of any application container's lifecycle.
sequenceDiagram
    participant K as kubelet
    participant CR as containerd
    participant CNI as CNI plugin

    K->>CR: RunPodSandbox (pause container)
    CR->>CNI: allocate IP, wire network namespace
    CNI-->>CR: pod IP 10.244.2.17
    K->>CR: PullImage (per imagePullPolicy)
    K->>CR: CreateContainer + StartContainer (init containers, in order)
    K->>CR: CreateContainer + StartContainer (app containers)
    K->>K: start probes, report status Running

That sequence is also the failure map for pods stuck in ContainerCreating: sandbox/CNI errors (no IP available, CNI plugin unhealthy), image pull failures, or volume mounts that won't attach -- kubectl describe pod events tell you which step stalled.

Image pulls

  • imagePullPolicy: IfNotPresent uses the node's local image cache; Always re-resolves the tag on every pod start (a digest check -- unchanged layers are not re-downloaded). Always is forced by default when the tag is latest.
  • Pulls happen per node. A 5GB image on a 100-node DaemonSet is 100 pulls -- this is why image size shows up as node-scaling latency, and why scale-sensitive platforms pre-pull or stream images.
  • Credentials come from imagePullSecrets or node-level credential providers; ErrImagePull → ImagePullBackOff is the retry loop in Troubleshooting.

Heartbeats: how the cluster knows a node is alive

Two channels, deliberately different in weight:

  • Lease renewal (kube-node-lease namespace, every ~10s): the cheap "I'm alive" signal. This is what the namespaces page alludes to with kube-node-lease.
  • Node status updates (every 5 minutes, or on change): the expensive full report -- capacity, conditions (Ready, MemoryPressure, DiskPressure, PIDPressure), images, versions.

If leases stop renewing (~40s grace), the node controller marks the node NotReady and, after the eviction grace period, the NoExecute taint machinery described in Scheduling and Placement begins moving pods away. Note the epistemology: the control plane never knows a node is dead, only silent -- which is exactly why StatefulSets refuse to auto-replace pods on unreachable nodes.

The kubelet as node janitor

Quietly, the kubelet also:

  • Garbage-collects images when disk crosses thresholds (defaults: start freeing at 85% usage, stop at 80%), oldest-unused first. Aggressive Always pulls plus tight disks means constant GC churn.
  • Garbage-collects dead containers, keeping the most recent for kubectl logs --previous.
  • Defends the node via node-pressure eviction: under memory/disk/PID pressure it evicts pods -- BestEffort first, then Burstable over requests -- and taints the node so the scheduler stops adding load. This is the node-local counterpart to the scheduler's admission-time math, covered in depth in the node-pressure eviction deep dive.
  • Runs static pods from /etc/kubernetes/manifests/ -- the bootstrapping mechanism behind the control plane itself.

When the kubelet itself breaks

The kubelet is the highest-value single process to understand for node debugging:

  • kubelet down: running containers keep running (the shim design guarantees it), but the node goes NotReady, probes stop, and no pod changes happen on that node. Fix: systemctl status kubelet, journalctl -u kubelet, commonly a bad config file, expired certificates, or swap enabled.
  • runtime down, kubelet up: node reports Ready=False with runtime errors; existing processes may survive, but nothing can start. Check systemctl status containerd and crictl ps (crictl speaks CRI directly -- it works when kubectl can't).
  • The node's view wins. Pod status in the API is the kubelet's report, not the truth itself. When API and reality disagree (a node partitioned from the control plane), the processes on the node are the reality; the API shows the last report plus Unknown.

Certification notes

  • CKA node-troubleshooting tasks are usually the kubelet: systemctl status kubelet, journalctl -u kubelet, config at /var/lib/kubelet/config.yaml, then systemctl daemon-reload && systemctl restart kubelet.
  • Know crictl ps -a / crictl logs for when the API server or kubectl is unavailable, and the static pod path /etc/kubernetes/manifests/.
  • Understand imagePullPolicy defaults (including the latest special case) -- a classic CKAD trip-up.

Key Takeaways

  • The kubelet is a reconciler scoped to one node: watch bound pods, converge the runtime, report status honestly.
  • Containers are ordinary kernel processes created through kubelet → CRI → containerd → shim → runc; the shim is why restarts of the management stack don't kill workloads.
  • The pause container physically holds the pod's shared namespaces -- it is the pod.
  • Liveness of nodes is inferred from cheap lease heartbeats; silence, not death, is what the control plane reacts to.
  • Kubelet down ≠ workloads down. Runtime layers fail independently, and crictl is your tool below the API.

Further Reading