Skip to content

etcd

Who this page is for: in plain English, etcd is the database where Kubernetes keeps everything - every object you've ever created lives there, and losing it means losing the cluster. This page is production depth, aimed at readers already comfortable with Pods, Services, and Deployments. New to Kubernetes? Start with the Beginner track and Control Plane first.

etcd is the single source of truth for a Kubernetes cluster. Every Deployment, Pod spec, Secret, ConfigMap, and piece of cluster state lives in etcd as a key-value pair. No Kubernetes component talks to etcd directly except the API server - the scheduler, controllers, and kubelets all go through the API server, which is the only client etcd needs to trust.

That single-writer-path design is deliberate: it means etcd itself can stay a generic, boring, consistent key-value store, while all Kubernetes-specific semantics (resource versioning, admission, RBAC, watches on typed resources) live in the API server layer above it. etcd graduated from the CNCF in 2020 and is used far beyond Kubernetes, but Kubernetes is by far its most consequential deployment.

Where etcd sits

flowchart TD
    Kubelet[kubelets] -->|watch/patch| API[kube-apiserver]
    Scheduler[kube-scheduler] -->|watch/patch| API
    CM[controller-manager] -->|watch/patch| API
    Users[kubectl / clients] -->|REST| API
    API -->|gRPC, mutual TLS\nONLY client| ETCD[(etcd cluster)]

    subgraph ETCD cluster: Raft consensus
        L[Leader]
        F1[Follower]
        F2[Follower]
        L <-->|replicate log| F1
        L <-->|replicate log| F2
    end

If etcd is unreachable, the API server can still serve reads from its watch cache for a short window, but all writes fail immediately and the cluster is effectively frozen for changes. If etcd loses quorum, the cluster is read-only at best - this is why etcd health is the top operational priority for any cluster operator, and why it's a core CKA topic.

Raft consensus, in the amount you actually need

etcd uses the Raft algorithm to keep multiple nodes in agreement about a single, ordered log of writes. You don't need to implement Raft to operate etcd well, but you do need its two operational consequences.

Leader election

One member is elected leader; all writes go through the leader, which replicates them to followers and only commits (acknowledges) a write once a majority of members have persisted it to their write-ahead log. Followers redirect client writes to the leader transparently. If the leader stops heartbeating, the remaining members hold an election and pick a new one - this takes a brief window (governed by --election-timeout, default 1000ms) during which the cluster cannot accept writes.

Quorum math - why odd numbers

A cluster of N members tolerates the loss of (N-1)/2 members while still having a majority available to elect a leader and commit writes.

Members Quorum needed Failures tolerated
1 1 0
2 2 0
3 2 1
4 3 1
5 3 2
6 4 2
7 4 3

Note that 4 members tolerate exactly the same number of failures as 3 (one), while requiring an extra machine, more disk, and more network traffic for replication - and paying a higher quorum size (3 vs 2) makes it more likely a network partition costs you quorum. This is why production etcd clusters run 3 or 5 members, never an even number: even counts add cost and fragility without adding fault tolerance. 3 members is the standard for most clusters; 5 is used when you want to tolerate two simultaneous failures (e.g., during a rolling upgrade combined with an unplanned outage).

Data model: flat keys, MVCC, and revisions

etcd stores a flat keyspace of byte-string keys to byte-string values - there's no schema, no tables, no nested structure at the storage layer. Kubernetes builds hierarchy on top by using structured key paths like:

/registry/pods/default/my-app-7d4f9c-x2b1p
/registry/deployments/production/api-server
/registry/secrets/default/db-credentials

Every write in etcd creates a new revision rather than overwriting in place - this is Multi-Version Concurrency Control (MVCC). etcd retains historical revisions (subject to compaction) so a client can read "the state as of revision N," which is exactly what Kubernetes' resourceVersion field maps to.

Why this matters for watches

Kubernetes' watch/informer machinery - the mechanism controllers use to react to changes without polling - is a direct reflection of etcd's own watch API. A client opens a watch on a key or key prefix starting at a given revision, and etcd streams every subsequent change as an event. The API server maintains long-lived watches against etcd on resource prefixes (/registry/pods, etc.) and re-multiplexes those events out to thousands of client watches (kubelets, controllers, kubectl get -w) without each of them opening a separate etcd watch. This is why a single etcd watch stream scales to a cluster with thousands of controllers, and also why a controller that mismanages its own watch/relist behavior can create real load on the API server (though rarely on etcd directly, since it's shielded by the API server's watch cache).

Compaction and defrag are different things

  • Compaction removes old MVCC revisions to bound the logical size of the keyspace history. Kubernetes' API server automatically compacts etcd periodically (default every 5 minutes, keeping recent history for watches to resume from).
  • Defragmentation reclaims the disk space and fixes internal fragmentation left behind after compaction - compaction alone does not shrink the on-disk database file. These are covered separately below because confusing them is a common cause of "why is my db file still huge after compaction" confusion.

Cluster topology: stacked vs external

flowchart LR
    subgraph Stacked
        N1[Node 1\nAPI server + etcd]
        N2[Node 2\nAPI server + etcd]
        N3[Node 3\nAPI server + etcd]
    end
    subgraph External
        A1[Node 1\nAPI server] -.-> E1[(etcd 1)]
        A2[Node 2\nAPI server] -.-> E2[(etcd 2)]
        A3[Node 3\nAPI server] -.-> E3[(etcd 3)]
    end

Stacked etcd - etcd runs as a static pod alongside the API server on each control plane node (the kubeadm default). Fewer machines to manage, but losing a control plane node loses both an API server and an etcd member simultaneously, coupling the two failure domains.

External etcd - etcd runs on its own dedicated machines, separate from API server nodes. More machines, more operational surface, but etcd and API-server failures are decoupled, and you can size/tune etcd hardware (fast NVMe, dedicated network) independently of API server compute needs. Preferred for large or latency-sensitive clusters.

Performance: why disk latency is the whole game

etcd's write path is: client request → Raft proposal → append to the write-ahead log (WAL) on disk → fsync → replicate to followers → commit once a majority have fsynced → apply to the in-memory/backend store. The fsync call is synchronous and on the critical path of every write - etcd cannot acknowledge a write until the WAL entry is durably on disk.

This makes etcd unusually sensitive to disk latency, not throughput. A slow or contended disk (network-attached storage with poor IOPS, a disk shared with other noisy workloads, spinning disks) directly increases write latency for every Kubernetes API write, and under sustained pressure can trigger leader elections (a leader that can't fsync fast enough stops heartbeating in time). etcd's own health checks surface this as elevated wal_fsync_duration_seconds.

Guidance from the etcd project and Kubernetes docs: use local SSD/NVMe storage, keep the WAL on its own disk/volume when possible, and avoid running etcd on the same disk as heavy I/O neighbors (container image layers, logs).

Sizing and quota

etcd defaults to a 2GB storage quota (--quota-backend-bytes, default 2*1024*1024*1024 bytes). The etcd project documents 8GB as the suggested practical maximum before performance degrades - etcd itself warns at startup if you configure a quota above that - so many production setups (including some Kubernetes distributions) raise the quota explicitly, but 8GB is a recommended ceiling, not the out-of-the-box default. When the backend database approaches the quota, etcd stops accepting writes and enters an alarm state (NOSPACE) until space is reclaimed via compaction + defragmentation.

# Check current DB size and alarm state
etcdctl endpoint status --write-out=table
etcdctl alarm list

# If NOSPACE is set, compact then defrag, then disarm
etcdctl compact $(etcdctl endpoint status --write-out="json" | python3 -c "import sys,json;print(json.load(sys.stdin)[0]['Status']['header']['revision'])")
etcdctl defrag --cluster
etcdctl alarm disarm

Defrag briefly blocks the member being defragmented - run it against one member at a time in a multi-member cluster (--cluster handles sequencing), never all at once, or you risk a simultaneous quorum-affecting stall.

Backup and restore: etcdctl snapshot

This is the single most important operational skill for etcd - treat it as muscle memory, not documentation you read during an incident. (The commands below set ETCDCTL_API=3 explicitly for clarity and to match older docs/scripts you'll encounter in the wild; it's been the default since etcd 3.4, so on any currently-supported etcd version - 3.5.x or 3.6.x - you can omit it.)

Taking a snapshot

ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-snapshot-$(date +%Y%m%d%H%M%S).db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Verify it's valid and check the revision/hash
ETCDCTL_API=3 etcdctl snapshot status /backup/etcd-snapshot-20260821120000.db --write-out=table

Take snapshots on a schedule (cron or a CronJob with node access), and always take one immediately before a Kubernetes version upgrade or any etcd cluster change.

Restoring a snapshot

Restore creates a new data directory from the snapshot - it does not restore in place, and it happens with etcd stopped:

# On each etcd member being restored:
sudo systemctl stop etcd    # or delete the static pod manifest to stop kubelet from running it

ETCDCTL_API=3 etcdctl snapshot restore /backup/etcd-snapshot-20260821120000.db \
  --name etcd-member-1 \
  --initial-cluster "etcd-member-1=https://10.0.0.1:2380,etcd-member-2=https://10.0.0.2:2380,etcd-member-3=https://10.0.0.3:2380" \
  --initial-cluster-token new-etcd-cluster \
  --initial-advertise-peer-urls https://10.0.0.1:2380 \
  --data-dir /var/lib/etcd-restored

# Point etcd's --data-dir at the new directory, then restart
sudo mv /var/lib/etcd /var/lib/etcd-old
sudo mv /var/lib/etcd-restored /var/lib/etcd
sudo systemctl start etcd   # or restore the static pod manifest

Every member must restore with matching --initial-cluster, --initial-cluster-token, and its own unique --name/peer URL, or the restored members won't agree they're forming the same new cluster. After restore, verify with etcdctl endpoint health and kubectl get nodes before declaring the cluster recovered.

kubeadm clusters additionally need the static pod manifest's --data-dir and hostPath volume updated to point at the restored directory if you didn't reuse the original path.

TLS requirements

etcd requires mutual TLS for both client and peer traffic in any production deployment:

  • Client TLS - authenticates the API server (and any etcdctl operator) to etcd. Certs typically at /etc/kubernetes/pki/etcd/{ca,server,healthcheck-client}.crt in kubeadm clusters.
  • Peer TLS - authenticates etcd members to each other for Raft replication traffic, using a separate peer CA/cert pair.
etcdctl member list \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/apiserver-etcd-client.crt \
  --key=/etc/kubernetes/pki/etcd/apiserver-etcd-client.key

Never run etcd with --client-transport-security=false in production - an unauthenticated etcd endpoint is equivalent to unauthenticated root access to the entire cluster's state, including all Secrets (which etcd stores base64-encoded, not encrypted, unless you've enabled encryption at rest via an EncryptionConfiguration).

Monitoring: the metrics that matter

etcd exposes Prometheus-format metrics that plug directly into a Prometheus stack:

Metric Watch for
etcd_server_has_leader drops to 0 during leader loss - writes are blocked
etcd_server_leader_changes_seen_total frequent increases indicate instability (disk, network, CPU pressure)
etcd_disk_wal_fsync_duration_seconds p99 should stay well under 10ms; sustained high values predict elections
etcd_disk_backend_commit_duration_seconds backend (bbolt) commit latency; same disk-pressure signal
etcd_server_proposals_failed_total failed Raft proposals - often quorum loss or leadership churn
etcd_mvcc_db_total_size_in_bytes approaching --quota-backend-bytes - schedule defrag
etcd_network_peer_round_trip_time_seconds inter-member network latency, especially across zones/regions

Common failure modes

Failure Symptom Cause / fix
Quorum loss API server writes hang or error; reads may still work briefly Lost majority of members (e.g., 2 of 3 down); restore quorum or recover from snapshot
Disk pressure / NOSPACE alarm Writes rejected cluster-wide DB hit quota-backend-bytes; compact + defrag, then alarm disarm
Slow disk causing election storms Frequent leader_changes_seen_total, API latency spikes Move etcd to dedicated fast SSD/NVMe; isolate from noisy neighbors
Watch stream overload High API server / etcd CPU, slow list-watches Too many controllers doing full relists instead of using bookmarks/incremental watches; review controller watch behavior
Split-brain risk from even member count Increased chance of losing quorum during partitions Always run 3 or 5 members, never 2 or 4
Clock skew across members Election instability, odd lease expirations Run NTP/chrony on all etcd hosts
Restoring into the wrong --initial-cluster config New cluster fails to form / members can't agree Ensure the restore command's cluster membership matches the target topology exactly