Cilium¶
Who this page is for: in plain English, Cilium is the plugin that gives pods their networking - routing, load balancing, and firewall rules - implemented in the Linux kernel with eBPF instead of iptables. This page is production depth, aimed at readers already comfortable with Pods, Services, and Deployments. New to Kubernetes? Start with the Beginner track and Networking Overview first.
Cilium is a CNI plugin that implements Kubernetes networking, network policy, and observability using eBPF (extended Berkeley Packet Filter). It runs programs directly in the Linux kernel without kernel modules or sidecars, giving it performance characteristics that iptables-based CNIs cannot match at scale.
A note on versions and stewardship: Cilium is a CNCF graduated project, originally created and still primarily driven by Isovalent, which Cisco acquired in 2024. The open source project is Apache 2.0 and CNCF-governed, but a commercial distribution (Isovalent Enterprise for Cilium, now sold through Cisco) exists alongside it and carries features that are not in the OSS build - so when evaluating a Cilium feature you read about, confirm it's in the open source project rather than the enterprise one. Cilium ships roughly two or three minor releases a year with a documented support window per release; the --version pinned in the install example below is illustrative, not a recommendation - check the stable releases page for the current line, and read the upgrade guide before any minor jump, since Cilium upgrades touch the datapath on every node.
Why eBPF changes the networking model¶
Traditional Kubernetes networking stacks traffic through iptables chains. At scale (thousands of services, tens of thousands of endpoints), iptables tables grow into the hundreds of thousands of rules. Rule evaluation is linear, rule changes require a full table rewrite, and the kernel has no visibility into what's happening.
eBPF programs attach to kernel hooks and execute in a sandboxed JIT-compiled environment. Cilium attaches to the network interface XDP hook for the fastest path, tc (traffic control) ingress/egress hooks for policy, and cgroup socket hooks for load balancing. Maps (hash tables, LRUs) replace iptables rules and update atomically in O(1).
Where eBPF programs actually attach¶
Cilium's datapath is built from several eBPF program types attached at different points in the packet's path through the kernel, each chosen for what it needs to see and how early it can act:
- XDP (eXpress Data Path) runs at the network driver level, before the kernel even allocates an
sk_bufffor the packet. This is the earliest possible interception point and the fastest - Cilium uses it for line-rate DDoS filtering and load-balancing decisions on ingress, when the NIC driver supports native XDP. - tc (traffic control) ingress/egress hooks run slightly later, once the packet is a normal kernel
sk_buff, and are where most of Cilium's policy enforcement, NAT, and routing decisions happen - both on the host-facing side of a veth pair and inside the pod's network namespace. - Socket-level hooks (
cgroup/connect4,cgroup/sendmsg4, and similar) attach at the socket layer via cgroups, interceptingconnect(),sendmsg(), andrecvmsg()syscalls before a packet is even constructed. This is what makes eBPF-based service load balancing so cheap: for a pod-to-service connection, Cilium rewrites the destination to a backend pod IP atconnect()time, so every subsequent packet on that socket is already addressed directly to the backend - no per-packet DNAT, no conntrack lookup on the hot path.
This is mechanically why Cilium bypasses iptables/netfilter: none of these programs touch the netfilter hook chain (PREROUTING, FORWARD, POSTROUTING, etc.) that iptables and its nftables successor rely on. Cilium's eBPF programs make forwarding and policy decisions directly against eBPF maps - O(1) hash lookups - instead of walking an ordered, linearly-evaluated rule list. The netfilter chain isn't disabled so much as never entered for pod traffic in kube-proxy-replacement mode; the kernel's iptables machinery still exists for anything Cilium doesn't manage (host firewalling, some NAT edge cases), but it's off the critical path for service routing and policy.
Architecture¶
flowchart TD
subgraph Node
Kubelet --> CiliumAgent["cilium-agent\n(DaemonSet)"]
CiliumAgent --> eBPF[eBPF programs\nloaded into kernel]
CiliumAgent --> Maps["eBPF maps\n(policy, endpoints,\nservices, NAT)"]
eBPF --> NIC[Network Interface]
end
subgraph Control
CiliumOperator["cilium-operator\n(Deployment)"] --> IPAM[IPAM pool\nmanagement]
CiliumOperator --> CRDs[CiliumNetworkPolicy\nCiliumClusterwideNetworkPolicy]
end
CiliumAgent <-- KV --> etcd[(etcd / CRDs)]
subgraph Hubble
HubbleRelay[hubble-relay] --> HubbleUI[hubble-ui]
CiliumAgent --> HubbleRelay
end
cilium-agent runs on every node. It watches Kubernetes API for endpoint, service, and policy changes, then compiles and loads eBPF programs and updates maps accordingly. No coordination with other nodes is needed for datapath decisions - the agent is fully self-contained.
cilium-operator handles cluster-wide operations: IPAM pool management, CRD cleanup, and leader-elected tasks that shouldn't run on every node.
Installation¶
helm repo add cilium https://helm.cilium.io/
# Pin an explicit version -- see the note on versions above for how to pick one
helm install cilium cilium/cilium --version <current-stable> \
--namespace kube-system \
--set kubeProxyReplacement=true \
--set k8sServiceHost=<API_SERVER_IP> \
--set k8sServicePort=6443 \
--set hubble.relay.enabled=true \
--set hubble.ui.enabled=true
kube-proxy replacement¶
Cilium can replace kube-proxy entirely. For in-cluster traffic, service load balancing moves to eBPF socket hooks - the kernel rewrites the destination address at connect() time, before a packet is ever built. That removes per-packet NAT and the iptables conntrack dependency for east-west traffic (Cilium tracks connections in its own eBPF maps), reduces latency, and handles very large endpoint counts without linear iptables rule evaluation.
There are two operational modes:
kubeProxyReplacement=true(full replacement): Cilium implements 100% of kube-proxy's functionality via eBPF and no kube-proxy runs at all. This is the mode that gets the full performance benefit and is the recommended path for new clusters.kubeProxyReplacement=partial(older Cilium versions used per-feature flags likehostServices/nodePort.enabled): Cilium handles some service types while kube-proxy remains for others. Mostly a migration aid; not where you want to end up long-term.
The key line to check in cilium status --verbose output is KubeProxyReplacement: True (or Strict in older CLI output) - anything else means Cilium is still deferring some service handling to kube-proxy or falling back. Confirm at the service level too:
kubectl -n kube-system exec ds/cilium -- cilium service list
kubectl -n kube-system exec ds/cilium -- cilium status --verbose | grep -A5 KubeProxyReplacement
The stale iptables trap¶
If a cluster previously ran kube-proxy and you enable Cilium's replacement without cleaning up, the old iptables KUBE-SERVICES and KUBE-NODEPORTS chains don't disappear on their own - kube-proxy has to be actually stopped and its rules flushed, not just have its DaemonSet scaled to zero while old rules linger from before. The symptoms are confusing because they're intermittent: some connections work (the ones Cilium's eBPF path claims first) while others hit stale, now-incorrect NAT rules pointing at pods that have since been rescheduled, producing connection resets or timeouts that look like flaky networking rather than a config problem. The fix is to explicitly remove kube-proxy (kubeadm clusters: delete the kube-proxy DaemonSet and run iptables-save | grep -i kube | iptables-restore style cleanup, or use cilium install's built-in kube-proxy-free bootstrapping on a fresh cluster) rather than assuming an absent DaemonSet means an absent ruleset.
Network policy¶
Cilium enforces standard NetworkPolicy objects, and extends them with CiliumNetworkPolicy for L3, L4, and L7 rules.
Standard NetworkPolicy (L3/L4)¶
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-frontend-to-api
namespace: production
spec:
podSelector:
matchLabels:
app: api
ingress:
- from:
- podSelector:
matchLabels:
app: frontend
ports:
- protocol: TCP
port: 8080
policyTypes:
- Ingress
CiliumNetworkPolicy - L7 HTTP¶
CiliumNetworkPolicy can enforce HTTP methods and paths, DNS FQDNs, Kafka topics, and gRPC services:
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: api-l7-policy
namespace: production
spec:
endpointSelector:
matchLabels:
app: api
ingress:
- fromEndpoints:
- matchLabels:
app: frontend
toPorts:
- ports:
- port: "8080"
protocol: TCP
rules:
http:
- method: "GET"
path: "/api/v1/.*"
- method: "POST"
path: "/api/v1/orders"
DNS-based egress policy¶
Restrict egress by FQDN instead of IP (IPs rotate; FQDNs are stable):
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: allow-stripe-egress
namespace: production
spec:
endpointSelector:
matchLabels:
app: payment
egress:
- toFQDNs:
- matchName: "api.stripe.com"
toPorts:
- ports:
- port: "443"
protocol: TCP
Cilium intercepts DNS responses and populates an internal map of FQDN-to-IP. Policy enforcement uses that map rather than requiring operators to track IP ranges.
Hubble - network observability¶
Hubble is Cilium's observability layer. It provides flow-level visibility into all traffic, surfaced as logs, metrics, and a UI - without sidecar injection.
Architecture: local agent, Relay, and UI¶
flowchart LR
subgraph "Node 1"
CA1["cilium-agent\n(local Hubble)"]
end
subgraph "Node 2"
CA2["cilium-agent\n(local Hubble)"]
end
subgraph "Node N"
CAN["cilium-agent\n(local Hubble)"]
end
CA1 --> Relay[hubble-relay]
CA2 --> Relay
CAN --> Relay
Relay --> CLI[hubble CLI]
Relay --> UI[hubble-ui]
Each cilium-agent runs an embedded Hubble instance that observes flows locally and keeps a bounded in-memory ring buffer of recent events for that node only - this is what powers hubble observe when run directly against a single node's agent, and it works even without Relay installed.
hubble-relay is a separate Deployment that connects to every agent's local Hubble gRPC API and aggregates flows across the whole cluster into a single stream. Without Relay, you'd have to query each node's agent individually and merge results yourself; Relay is what makes hubble observe --namespace production return cluster-wide results from one command.
hubble-ui is a web frontend that queries Relay and renders a live, graphical service dependency map - which services are talking to which, at what rate, and whether traffic is being allowed or dropped by policy. It's most useful for building an initial mental model of an unfamiliar cluster's traffic patterns, or for demonstrating to a security team what a NetworkPolicy would actually affect before applying it.
Filtering flows with hubble observe¶
cilium hubble enable
hubble observe --namespace production --follow
# Filter by identity - Cilium's numeric security identity, stable across pod restarts
hubble observe --identity 4520 --follow
# Filter by verdict: FORWARDED, DROPPED, ERROR, AUDIT
hubble observe --verdict DROPPED --follow
# Combine namespace, pod, and verdict filters
hubble observe --namespace production --pod frontend-7d9b84c4f-x2k4p --verdict DROPPED --follow
# Filter by L7 HTTP details once L7 visibility is enabled
hubble observe --namespace production --protocol http --http-status 500 --follow
# Filter by direction and port
hubble observe --to-port 443 --type trace:to-endpoint --follow
Hubble exports Prometheus metrics for request rates, drop rates, and latency by namespace, pod, and destination. These are the same signals you'd get from a service mesh without the sidecar overhead.
# Find which flows are being dropped by policy
hubble observe --verdict DROPPED -j | jq '{src: .source.namespace, dst: .destination.namespace, reason: .drop_reason_desc}'
From observed traffic to a network policy¶
One of Hubble's most practical uses is closing the loop between "what traffic actually happens" and "what policy should allow." Rather than authoring a CiliumNetworkPolicy from a guess about a service's dependencies, observe its real traffic first and derive the policy from what you see:
# Watch everything a workload sends and receives over a representative window
hubble observe --pod production/api --output json --follow > api-flows.json
Capture a representative window of FORWARDED flows for the workload (a full business cycle, not just a quiet minute), then build the CiliumNetworkPolicy rules directly from the distinct (source identity, destination identity, port, L7 method/path) tuples that appear in the JSON output - each unique combination becomes one explicit rule. Treat the result as a first draft, not a final answer - it reflects only the traffic that occurred during the observation window, so a service that talks to a dependency once a day (a batch job, a certificate renewal) can be silently missing from the policy unless the window is long enough to capture it. Review the generated rules and widen or narrow them deliberately rather than applying them blind.
Hubble UI provides a graphical service map that shows live traffic flows and policy decisions.
Encryption¶
Cilium supports transparent encryption of all node-to-node traffic with two options:
IPsec - kernel-native, highest compatibility, requires key rotation management:
helm upgrade cilium cilium/cilium --reuse-values \
--set encryption.enabled=true \
--set encryption.type=ipsec
WireGuard - simpler, better performance, requires kernel 5.6+:
helm upgrade cilium cilium/cilium --reuse-values \
--set encryption.enabled=true \
--set encryption.type=wireguard
Both modes encrypt all pod-to-pod traffic crossing node boundaries without application changes. WireGuard keys are automatically generated and rotated by Cilium.
Native mTLS with ztunnel (beta): recent Cilium releases can optionally use ztunnel - the same lightweight, per-node Rust proxy that powers Istio's ambient mesh - re-engineered to plug into Cilium's own control plane, with SPIRE as the certificate authority for workload identity. This is a joint effort between Isovalent and Microsoft: it brings connection-level mutual TLS into the datapath itself rather than relying solely on WireGuard/IPsec node-to-node tunnels, while keeping WireGuard/IPsec for non-TCP traffic and hitless upgrades. It's beta, with real limits at the time of writing - namespace-scoped enablement, TCP only, and no Cluster Mesh support when ztunnel encryption is on - so check the Cilium docs for which release it landed in and what's still unsupported before building on it.
Cluster Mesh¶
Cluster Mesh connects multiple Kubernetes clusters at the network level, allowing pods in one cluster to reach services in another using standard DNS and Kubernetes service names.
flowchart LR
subgraph Cluster A
PodA[Pod] --> SvcA[Service]
end
subgraph Cluster B
SvcB[Global Service\nmirror in A]
PodB[Pod]
end
SvcA --> SvcB --> PodB
cilium clustermesh enable --context cluster-a
cilium clustermesh enable --context cluster-b
cilium clustermesh connect --destination-context cluster-b
Annotate a service as global:
Global services load-balance across healthy endpoints in all connected clusters. Combined with service.cilium.io/affinity: "local", you get prefer-local-cluster behavior with automatic failover.
Since Cilium 1.16, Cluster Mesh runs with KVStoreMesh enabled by default: each cluster caches the state it learns from remote clusters in its own local kvstore instead of every agent watching every remote cluster directly. Local agents then sync only against their own cluster's cache, which cuts the number of remote watches an agent needs to hold open and improves both scalability and blast-radius isolation as the number of connected clusters grows.
Cilium as a service mesh¶
Cilium's CiliumEnvoyConfig and Cilium Service Mesh mode provide L7 traffic management (retries, timeouts, header manipulation, traffic splitting) via Envoy deployed as a per-node DaemonSet - not per-pod sidecars. This gives service mesh capabilities with a much lower resource footprint.
helm upgrade cilium cilium/cilium --reuse-values \
--set ingressController.enabled=true \
--set ingressController.loadbalancerMode=dedicated
Gateway API and Ingress¶
Because Cilium already runs an eBPF datapath and can deploy Envoy per-node, it can act as a full Kubernetes Ingress controller or Gateway API implementation, removing the need to run a separate ingress controller (nginx-ingress, HAProxy, etc.) alongside it.
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: public-gateway
spec:
gatewayClassName: cilium
listeners:
- name: https
port: 443
protocol: HTTPS
tls:
mode: Terminate
certificateRefs:
- name: my-cert-tls
The practical case for consolidating onto Cilium's ingress/Gateway API support: one less component to install, upgrade, and secure, one less set of iptables/proxy rules interacting with the CNI's own datapath, and ingress traffic gets the same Hubble observability as every other flow in the cluster. The tradeoff is ecosystem maturity - a dedicated ingress controller with a long track record may have edge-case features or a larger community knowledge base that a newer built-in controller hasn't accumulated yet, so evaluate feature parity for your specific ingress needs (WAF integrations, complex rewrite rules, etc.) before consolidating.
Bandwidth Manager and BBR¶
Bandwidth Manager uses eBPF traffic control (the same tc hooks used for policy enforcement) to enforce per-pod egress bandwidth limits and to enable fair queuing, avoiding the head-of-line blocking and bufferbloat that can occur when bursty pods share a node's network interface without any shaping.
helm upgrade cilium cilium/cilium --reuse-values \
--set bandwidthManager.enabled=true \
--set bandwidthManager.bbr=true
With bandwidthManager.bbr=true, Cilium enables the kernel's BBR (Bottleneck Bandwidth and RTT) congestion control algorithm for pod traffic instead of the default CUBIC. BBR models the actual bottleneck bandwidth and round-trip time rather than reacting to packet loss, which tends to perform meaningfully better on high-bandwidth, high-latency paths (cross-region traffic, paths with shallow buffers) where CUBIC's loss-based backoff leaves throughput on the table. This requires a kernel with BBR support compiled in (most modern distributions ship it).
Egress Gateway¶
Many enterprise environments firewall outbound traffic by source IP - a third-party API, a partner's on-prem system, or a legacy internal service will only accept connections from an allow-listed address. In a Kubernetes cluster, pod IPs are ephemeral and node IPs can be numerous, which makes a naive "allow-list the cluster" approach impractical.
Egress Gateway solves this by routing selected pods' egress traffic through one or a small set of designated gateway nodes, so external services see a small, stable set of source IPs regardless of which node a pod actually schedules on.
flowchart LR
PodA["Pod\n(node 3)"] -.-> GW["Egress Gateway node\n(stable IP: 203.0.113.10)"]
PodB["Pod\n(node 7)"] -.-> GW
GW --> Ext["External service\n(firewalled by source IP)"]
apiVersion: cilium.io/v2
kind: CiliumEgressGatewayPolicy
metadata:
name: partner-api-egress
spec:
selectors:
- podSelector:
matchLabels:
app: billing
destinationCIDRs:
- "198.51.100.0/24"
egressGateway:
nodeSelector:
matchLabels:
egress-gateway: "true"
egressIP: "203.0.113.10"
Traffic from matching pods to matching destination CIDRs is SNAT'd to the configured egressIP as it leaves the cluster, regardless of which node the pod is actually running on. This is one of the more common reasons enterprises adopt Cilium specifically - it solves a real, otherwise-painful firewall-allow-listing problem without requiring a NAT gateway appliance or cloud-specific egress IP feature.
BGP control plane¶
Cilium can advertise pod and service CIDRs to upstream BGP routers, enabling bare-metal clusters to expose LoadBalancer services without a cloud provider. Use the BGPv2 API (CiliumBGPClusterConfig, CiliumBGPPeerConfig, CiliumBGPAdvertisement, introduced in Cilium 1.16). The older CiliumBGPPeeringPolicy is deprecated and on a removal path, so clusters still running v1 peering-policy manifests need to migrate - because this gates an upgrade, confirm the exact deprecation and removal releases in the Cilium upgrade guide for your target version rather than taking a version number from this page:
apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata:
name: rack-bgp
spec:
nodeSelector:
matchLabels:
rack: "1"
bgpInstances:
- name: instance-65001
localASN: 65001
peers:
- name: tor-router
peerAddress: 10.0.0.1
peerASN: 65000
peerConfigRef:
name: peer-config
Troubleshooting playbook¶
Work through these in order when something in the datapath isn't behaving as expected - each tool narrows the problem to a smaller layer:
cilium status --verbose- the first stop for any issue. Confirms the agent is healthy, kube-proxy replacement is actually active, and the control-plane connection to the Kubernetes API is up. ACilium: WarningorDegradedstatus here usually points straight at the root cause.
cilium-health- tests actual node-to-node and pod-to-pod connectivity through the datapath, independent of any specific application. Useful for distinguishing "the network fabric itself is broken" from "policy is blocking this specific traffic":
cilium monitor- a live, low-level stream of datapath events (drops, policy verdicts, debug traces) directly from the eBPF programs, filterable by type. This is a level below Hubble - useful when Hubble itself isn't seeing what you expect, or for drops that occur before Hubble's own observation point:
kubectl -n kube-system exec ds/cilium -- cilium monitor --type drop
kubectl -n kube-system exec ds/cilium -- cilium monitor -v --related-to <endpoint-id>
- eBPF map pressure - every eBPF map (policy, connection tracking, NAT, load-balancing) has a fixed maximum size set at agent startup. Once a map fills, new entries fail to insert and you get silent packet drops or new connections failing while existing ones keep working - a classic "some things fail intermittently at scale" symptom. Check occupancy against configured limits before it becomes an outage:
kubectl -n kube-system exec ds/cilium -- cilium bpf policy list
kubectl -n kube-system exec ds/cilium -- cilium bpf nat list
kubectl -n kube-system exec ds/cilium -- cilium bpf ct list global | wc -l
kubectl -n kube-system exec ds/cilium -- cilium map get cilium_ct_any4_global # shows max entries vs current
Map sizes are tunable via Helm values (e.g. bpf.ctTcpMax, bpf.natMax) - on very large or very high-connection-churn clusters, size them proactively rather than reactively once drops start.
Operational patterns¶
Debug policy drops: hubble observe --verdict DROPPED is far faster than reading iptables logs. Every drop includes the policy name and direction that blocked it.
Identity-based policy: Cilium assigns a numeric identity to each endpoint based on its labels. Policy evaluation uses identities, not IPs. This means policy is stable across pod restarts and IP reassignments.
Monitor eBPF map pressure: cilium bpf policy list and cilium bpf nat list show map occupancy. At very high scale, tune map sizes before running into hard limits.
Bandwidth management: Cilium supports egress bandwidth limits via eBPF traffic control, without requiring a separate rate-limiting sidecar.
Common mistakes¶
Leaving kube-proxy's iptables rules in place after enabling Cilium's kube-proxy replacement. Scaling the kube-proxy DaemonSet to zero doesn't remove its rules. Stale KUBE-SERVICES/KUBE-NODEPORTS chains produce intermittent, hard-to-diagnose connection failures that look like flaky infrastructure. Explicitly clean up old rules, or start from a cluster that never ran kube-proxy.
Writing CiliumNetworkPolicy egress rules against pod or node IPs instead of toEndpoints selectors or toFQDNs. IPs are ephemeral in Kubernetes; a policy pinned to an IP silently stops matching the moment a pod is rescheduled or a backend rotates. Use label-based toEndpoints for in-cluster destinations and toFQDNs for external ones.
Deriving a CiliumNetworkPolicy from too short a Hubble observation window. A policy built from five minutes of traffic will miss anything that happens on a longer cycle - nightly batch jobs, periodic health checks to a third party, cert renewal callbacks - and those dependencies will start failing silently the moment the policy goes into enforcing mode.
Ignoring maxEjectionPercent-equivalent safety limits when sizing eBPF maps. Sizing connection-tracking or NAT maps too small for actual cluster scale causes drops that look like application bugs; the fix is capacity planning based on real endpoint and connection counts, not the defaults from a much smaller test cluster.
Assuming Hubble UI's service map is a complete picture without checking the observation time range. Like the policy-generation flow above, the UI only shows what happened during the window it queried - a service that looks isolated may simply not have talked to anything during that window.
Source Links¶
- Cilium documentation
- cilium/cilium on GitHub and its releases
- Cilium upgrade guide (deprecations and removals per release)
- kube-proxy replacement
- CiliumNetworkPolicy reference
- Cilium BGP control plane
- Cluster Mesh
- Transparent encryption (WireGuard / IPsec)
- Hubble
- Cilium Gateway API support
- CNCF project page: Cilium
Related Concepts¶
- Istio - the sidecar/ambient mesh Cilium's service-mesh mode is usually compared against
- Linkerd - the lightweight mesh alternative
- Network Policies - the upstream API
CiliumNetworkPolicyextends - Networking Overview - how the CNI fits into the wider networking model
- Gateway API - Cilium is a conformant Gateway API implementation