Control Plane and etcd¶

The control plane is the set of components that decide what should happen in the cluster. This page explains how those components actually coordinate, what breaks when each one fails, and why the answer to "the control plane is down!" is usually calmer than people expect.

The cast, and what each one actually owns¶

The single most important operational fact: the control plane controls change, not existence. Every workload decision already made is executed by kubelets and survives on the nodes. If the entire control plane vanishes, running pods keep running, Services keep routing (the rules are already programmed on every node), and DNS keeps answering. What you lose is the ability to change anything -- and self-healing, which is a form of change.

etcd: the only stateful thing in the room¶

Everything else in the control plane is stateless and disposable. etcd is where the cluster actually is -- every object you have ever applied lives there as a key like /registry/deployments/production/web.

etcd is a Raft-based consensus store, which buys three properties Kubernetes depends on:

• Linearizable writes: every write is agreed on by a majority (quorum) of members before it is acknowledged. There is exactly one truth, even during network partitions.
• Watch streams: clients can subscribe to changes from a specific revision. This is the foundation the entire watch mechanism is built on.
• MVCC history: etcd keeps a revision history, which is what makes "resume the watch from where I left off" possible.

Quorum arithmetic you must internalize¶

A cluster of N members tolerates floor((N-1)/2) failures:

Two facts follow that regularly surprise people:

• Even numbers add nothing. Four members still only tolerate one failure (quorum is 3) -- you added a machine and gained risk, not safety. Run 3 or 5.
• Losing quorum stops writes, not reads. With 1 of 3 members alive, etcd can still serve (possibly stale) reads, but nothing can be written -- which means no pod can be created, updated, or deleted cluster-wide. The fix is restoring members, or in a disaster, restoring from snapshot.

etcd is also latency-sensitive: every write waits for a disk fsync on a majority of members. Slow disks (or etcd sharing a disk with something noisy) surface as cluster-wide API slowness that looks like an API server problem. This is why every serious deployment puts etcd on dedicated SSD/NVMe storage.

The API server: stateless by design¶

The API server holds no state of its own -- you can run several behind a load balancer and kill any of them freely. Its job is the request pipeline (authn → authz → admission → persist) plus one big performance trick: the watch cache. The API server maintains an in-memory, watch-updated copy of etcd state and serves most reads and all watch streams from it, shielding etcd from the thundering herd of kubelets and controllers.

One leader per controller, many replicas¶

Control loops must not run twice -- two ReplicaSet controllers would double-create pods. But you also want multiple controller-manager and scheduler replicas for availability. Kubernetes resolves this with leader election via Lease objects:

sequenceDiagram
    participant A as controller-manager A
    participant L as Lease (kube-system)
    participant B as controller-manager B

    A->>L: acquire lease (holder: A, renew every 10s)
    B->>L: try acquire -- held by A, stand by
    Note over A: A does all the work
    A--xL: A crashes, stops renewing
    Note over L: lease expires (~15s)
    B->>L: acquire lease (holder: B)
    Note over B: B takes over all loops

All replicas run, but only the lease holder acts; the others idle as hot standbys. You can see this live:

kubectl get leases -n kube-system

This is the same pattern operators and other controllers use, and it's a general distributed-systems lesson: Kubernetes prefers at-most-one active with fast failover over trying to make concurrent work safe.

Who runs the control plane? (the bootstrapping trick)¶

In kubeadm clusters, the control plane runs as static pods: the kubelet on each control-plane node reads manifests directly from /etc/kubernetes/manifests/ and runs them without asking the API server -- which is how the API server can be run by Kubernetes machinery before the API exists. The API server then shows read-only "mirror pods" for them, which is why you can kubectl get pod kube-apiserver-node1 -n kube-system but not meaningfully edit it; the file on disk is the truth.

This design explains a classic troubleshooting move: if the API server is down, kubectl is useless -- you fix the control plane by editing manifests in /etc/kubernetes/manifests/ and reading container state with crictl, because the kubelet is still watching that directory. (Managed platforms like EKS/GKE/AKS hide all of this -- the control plane is the provider's problem, which is most of what you're paying for.)

HA topologies¶

graph TB
    subgraph Stacked
        direction TB
        N1[node 1\napiserver + etcd]
        N2[node 2\napiserver + etcd]
        N3[node 3\napiserver + etcd]
    end
    subgraph External etcd
        direction TB
        A1[apiserver nodes ×2..3]
        E1[(etcd cluster ×3\ndedicated)]
        A1 --> E1
    end
    LB[Load balancer\napi.cluster.example:6443] --> N1 & N2 & N3

• Stacked (etcd co-located with each control-plane node): simpler, fewer machines, the kubeadm default. Losing a node costs you both an API server and an etcd member at once.
• External etcd: isolates etcd's failure domain and disk I/O from the API servers at the cost of more machines and more operational surface.

Either way, kubelets and clients reach the API through a load-balanced endpoint, never a specific node.

Failure modes, ranked by how scared you should be¶

1. Scheduler or controller-manager down: lowest severity. Existing workloads unaffected; new pods pend and healing pauses. Standby replicas usually take over in seconds.
2. API server down: no reads or writes, kubelets can't report status, kubectl dead -- but workloads, Services, and DNS keep functioning on the nodes. Fix the API server; don't touch the nodes.
3. etcd quorum lost: the cluster is frozen. Still serving traffic, but incapable of change and incapable of healing. Restore members or restore from snapshot -- this is why the etcd backup drill is non-negotiable.
4. etcd data lost, no backup: the only truly unrecoverable failure in Kubernetes. Every other component is rebuildable from etcd; nothing rebuilds etcd.

That ranking is worth restating as the core mental model: Kubernetes degrades from "self-driving" to "cruise control" to "hands on wheel" -- it does not fall off a cliff.

Certification notes¶

• CKA expects you to know the static pod manifest path (/etc/kubernetes/manifests/), that the kubelet watches it, and how to use crictl ps / crictl logs when the API server is down.
• etcd snapshot save/restore with full TLS flags is a near-guaranteed task -- practice it until it's muscle memory (see Maintenance).
• Know the component-failure symptoms table above; "pods stay Pending, why?" with a stopped scheduler is a classic scenario.

Key Takeaways¶

• The control plane controls change; node-level components keep everything already decided running without it.
• etcd is the only stateful component -- quorum governs writes, odd member counts only, disk latency is cluster latency.
• Stateless API servers scale out; singleton control loops use Lease-based leader election for at-most-one-active.
• Static pods solve the bootstrapping paradox and define how you repair a broken control plane.
• Back up etcd. Everything else is replaceable.

Related Concepts¶

• Kubernetes API -- the request pipeline and watch mechanism
• Kubelet and Container Runtime -- the node-side counterpart
• Kubernetes Overview -- where these components fit
• Maintenance -- etcd backup/restore and upgrades
• Scheduling and Placement -- what the scheduler does with its lease

Further Reading¶

• etcd Consistency and the Kubernetes Control Plane
• Making etcd Incidents Easier to Debug