Skip to content

Cilium

Who this page is for: in plain English, Cilium is the plugin that gives pods their networking - routing, load balancing, and firewall rules - implemented in the Linux kernel with eBPF instead of iptables. This page is production depth, aimed at readers already comfortable with Pods, Services, and Deployments. New to Kubernetes? Start with the Beginner track and Networking Overview first.

Cilium is a CNI plugin that implements Kubernetes networking, network policy, and observability using eBPF (extended Berkeley Packet Filter). It runs programs directly in the Linux kernel without kernel modules or sidecars, giving it performance characteristics that iptables-based CNIs cannot match at scale.

A note on versions and stewardship: Cilium is a CNCF graduated project, originally created and still primarily driven by Isovalent, which Cisco acquired in 2024. The open source project is Apache 2.0 and CNCF-governed, but a commercial distribution (Isovalent Enterprise for Cilium, now sold through Cisco) exists alongside it and carries features that are not in the OSS build - so when evaluating a Cilium feature you read about, confirm it's in the open source project rather than the enterprise one. Cilium ships roughly two or three minor releases a year with a documented support window per release; the --version pinned in the install example below is illustrative, not a recommendation - check the stable releases page for the current line, and read the upgrade guide before any minor jump, since Cilium upgrades touch the datapath on every node.

Why eBPF changes the networking model

Traditional Kubernetes networking stacks traffic through iptables chains. At scale (thousands of services, tens of thousands of endpoints), iptables tables grow into the hundreds of thousands of rules. Rule evaluation is linear, rule changes require a full table rewrite, and the kernel has no visibility into what's happening.

eBPF programs attach to kernel hooks and execute in a sandboxed JIT-compiled environment. Cilium attaches to the network interface XDP hook for the fastest path, tc (traffic control) ingress/egress hooks for policy, and cgroup socket hooks for load balancing. Maps (hash tables, LRUs) replace iptables rules and update atomically in O(1).

Where eBPF programs actually attach

Cilium's datapath is built from several eBPF program types attached at different points in the packet's path through the kernel, each chosen for what it needs to see and how early it can act:

  • XDP (eXpress Data Path) runs at the network driver level, before the kernel even allocates an sk_buff for the packet. This is the earliest possible interception point and the fastest - Cilium uses it for line-rate DDoS filtering and load-balancing decisions on ingress, when the NIC driver supports native XDP.
  • tc (traffic control) ingress/egress hooks run slightly later, once the packet is a normal kernel sk_buff, and are where most of Cilium's policy enforcement, NAT, and routing decisions happen - both on the host-facing side of a veth pair and inside the pod's network namespace.
  • Socket-level hooks (cgroup/connect4, cgroup/sendmsg4, and similar) attach at the socket layer via cgroups, intercepting connect(), sendmsg(), and recvmsg() syscalls before a packet is even constructed. This is what makes eBPF-based service load balancing so cheap: for a pod-to-service connection, Cilium rewrites the destination to a backend pod IP at connect() time, so every subsequent packet on that socket is already addressed directly to the backend - no per-packet DNAT, no conntrack lookup on the hot path.

This is mechanically why Cilium bypasses iptables/netfilter: none of these programs touch the netfilter hook chain (PREROUTING, FORWARD, POSTROUTING, etc.) that iptables and its nftables successor rely on. Cilium's eBPF programs make forwarding and policy decisions directly against eBPF maps - O(1) hash lookups - instead of walking an ordered, linearly-evaluated rule list. The netfilter chain isn't disabled so much as never entered for pod traffic in kube-proxy-replacement mode; the kernel's iptables machinery still exists for anything Cilium doesn't manage (host firewalling, some NAT edge cases), but it's off the critical path for service routing and policy.

Architecture

flowchart TD
    subgraph Node
        Kubelet --> CiliumAgent["cilium-agent\n(DaemonSet)"]
        CiliumAgent --> eBPF[eBPF programs\nloaded into kernel]
        CiliumAgent --> Maps["eBPF maps\n(policy, endpoints,\nservices, NAT)"]
        eBPF --> NIC[Network Interface]
    end
    subgraph Control
        CiliumOperator["cilium-operator\n(Deployment)"] --> IPAM[IPAM pool\nmanagement]
        CiliumOperator --> CRDs[CiliumNetworkPolicy\nCiliumClusterwideNetworkPolicy]
    end
    CiliumAgent <-- KV --> etcd[(etcd / CRDs)]
    subgraph Hubble
        HubbleRelay[hubble-relay] --> HubbleUI[hubble-ui]
        CiliumAgent --> HubbleRelay
    end

cilium-agent runs on every node. It watches Kubernetes API for endpoint, service, and policy changes, then compiles and loads eBPF programs and updates maps accordingly. No coordination with other nodes is needed for datapath decisions - the agent is fully self-contained.

cilium-operator handles cluster-wide operations: IPAM pool management, CRD cleanup, and leader-elected tasks that shouldn't run on every node.

Installation

helm repo add cilium https://helm.cilium.io/

# Pin an explicit version -- see the note on versions above for how to pick one
helm install cilium cilium/cilium --version <current-stable> \
  --namespace kube-system \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=<API_SERVER_IP> \
  --set k8sServicePort=6443 \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true

kube-proxy replacement

Cilium can replace kube-proxy entirely. For in-cluster traffic, service load balancing moves to eBPF socket hooks - the kernel rewrites the destination address at connect() time, before a packet is ever built. That removes per-packet NAT and the iptables conntrack dependency for east-west traffic (Cilium tracks connections in its own eBPF maps), reduces latency, and handles very large endpoint counts without linear iptables rule evaluation.

There are two operational modes:

  • kubeProxyReplacement=true (full replacement): Cilium implements 100% of kube-proxy's functionality via eBPF and no kube-proxy runs at all. This is the mode that gets the full performance benefit and is the recommended path for new clusters.
  • kubeProxyReplacement=partial (older Cilium versions used per-feature flags like hostServices/nodePort.enabled): Cilium handles some service types while kube-proxy remains for others. Mostly a migration aid; not where you want to end up long-term.
# Verify kube-proxy is absent and Cilium handles services
cilium status --verbose

The key line to check in cilium status --verbose output is KubeProxyReplacement: True (or Strict in older CLI output) - anything else means Cilium is still deferring some service handling to kube-proxy or falling back. Confirm at the service level too:

kubectl -n kube-system exec ds/cilium -- cilium service list
kubectl -n kube-system exec ds/cilium -- cilium status --verbose | grep -A5 KubeProxyReplacement

The stale iptables trap

If a cluster previously ran kube-proxy and you enable Cilium's replacement without cleaning up, the old iptables KUBE-SERVICES and KUBE-NODEPORTS chains don't disappear on their own - kube-proxy has to be actually stopped and its rules flushed, not just have its DaemonSet scaled to zero while old rules linger from before. The symptoms are confusing because they're intermittent: some connections work (the ones Cilium's eBPF path claims first) while others hit stale, now-incorrect NAT rules pointing at pods that have since been rescheduled, producing connection resets or timeouts that look like flaky networking rather than a config problem. The fix is to explicitly remove kube-proxy (kubeadm clusters: delete the kube-proxy DaemonSet and run iptables-save | grep -i kube | iptables-restore style cleanup, or use cilium install's built-in kube-proxy-free bootstrapping on a fresh cluster) rather than assuming an absent DaemonSet means an absent ruleset.

Network policy

Cilium enforces standard NetworkPolicy objects, and extends them with CiliumNetworkPolicy for L3, L4, and L7 rules.

Standard NetworkPolicy (L3/L4)

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-api
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: api
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend
      ports:
        - protocol: TCP
          port: 8080
  policyTypes:
    - Ingress

CiliumNetworkPolicy - L7 HTTP

CiliumNetworkPolicy can enforce HTTP methods and paths, DNS FQDNs, Kafka topics, and gRPC services:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: api-l7-policy
  namespace: production
spec:
  endpointSelector:
    matchLabels:
      app: api
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: frontend
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP
          rules:
            http:
              - method: "GET"
                path: "/api/v1/.*"
              - method: "POST"
                path: "/api/v1/orders"

DNS-based egress policy

Restrict egress by FQDN instead of IP (IPs rotate; FQDNs are stable):

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-stripe-egress
  namespace: production
spec:
  endpointSelector:
    matchLabels:
      app: payment
  egress:
    - toFQDNs:
        - matchName: "api.stripe.com"
      toPorts:
        - ports:
            - port: "443"
              protocol: TCP

Cilium intercepts DNS responses and populates an internal map of FQDN-to-IP. Policy enforcement uses that map rather than requiring operators to track IP ranges.

Hubble - network observability

Hubble is Cilium's observability layer. It provides flow-level visibility into all traffic, surfaced as logs, metrics, and a UI - without sidecar injection.

Architecture: local agent, Relay, and UI

flowchart LR
    subgraph "Node 1"
        CA1["cilium-agent\n(local Hubble)"]
    end
    subgraph "Node 2"
        CA2["cilium-agent\n(local Hubble)"]
    end
    subgraph "Node N"
        CAN["cilium-agent\n(local Hubble)"]
    end
    CA1 --> Relay[hubble-relay]
    CA2 --> Relay
    CAN --> Relay
    Relay --> CLI[hubble CLI]
    Relay --> UI[hubble-ui]

Each cilium-agent runs an embedded Hubble instance that observes flows locally and keeps a bounded in-memory ring buffer of recent events for that node only - this is what powers hubble observe when run directly against a single node's agent, and it works even without Relay installed.

hubble-relay is a separate Deployment that connects to every agent's local Hubble gRPC API and aggregates flows across the whole cluster into a single stream. Without Relay, you'd have to query each node's agent individually and merge results yourself; Relay is what makes hubble observe --namespace production return cluster-wide results from one command.

hubble-ui is a web frontend that queries Relay and renders a live, graphical service dependency map - which services are talking to which, at what rate, and whether traffic is being allowed or dropped by policy. It's most useful for building an initial mental model of an unfamiliar cluster's traffic patterns, or for demonstrating to a security team what a NetworkPolicy would actually affect before applying it.

Filtering flows with hubble observe

cilium hubble enable
hubble observe --namespace production --follow

# Filter by identity - Cilium's numeric security identity, stable across pod restarts
hubble observe --identity 4520 --follow

# Filter by verdict: FORWARDED, DROPPED, ERROR, AUDIT
hubble observe --verdict DROPPED --follow

# Combine namespace, pod, and verdict filters
hubble observe --namespace production --pod frontend-7d9b84c4f-x2k4p --verdict DROPPED --follow

# Filter by L7 HTTP details once L7 visibility is enabled
hubble observe --namespace production --protocol http --http-status 500 --follow

# Filter by direction and port
hubble observe --to-port 443 --type trace:to-endpoint --follow

Hubble exports Prometheus metrics for request rates, drop rates, and latency by namespace, pod, and destination. These are the same signals you'd get from a service mesh without the sidecar overhead.

# Find which flows are being dropped by policy
hubble observe --verdict DROPPED -j | jq '{src: .source.namespace, dst: .destination.namespace, reason: .drop_reason_desc}'

From observed traffic to a network policy

One of Hubble's most practical uses is closing the loop between "what traffic actually happens" and "what policy should allow." Rather than authoring a CiliumNetworkPolicy from a guess about a service's dependencies, observe its real traffic first and derive the policy from what you see:

# Watch everything a workload sends and receives over a representative window
hubble observe --pod production/api --output json --follow > api-flows.json

Capture a representative window of FORWARDED flows for the workload (a full business cycle, not just a quiet minute), then build the CiliumNetworkPolicy rules directly from the distinct (source identity, destination identity, port, L7 method/path) tuples that appear in the JSON output - each unique combination becomes one explicit rule. Treat the result as a first draft, not a final answer - it reflects only the traffic that occurred during the observation window, so a service that talks to a dependency once a day (a batch job, a certificate renewal) can be silently missing from the policy unless the window is long enough to capture it. Review the generated rules and widen or narrow them deliberately rather than applying them blind.

Hubble UI provides a graphical service map that shows live traffic flows and policy decisions.

Encryption

Cilium supports transparent encryption of all node-to-node traffic with two options:

IPsec - kernel-native, highest compatibility, requires key rotation management:

helm upgrade cilium cilium/cilium --reuse-values \
  --set encryption.enabled=true \
  --set encryption.type=ipsec

WireGuard - simpler, better performance, requires kernel 5.6+:

helm upgrade cilium cilium/cilium --reuse-values \
  --set encryption.enabled=true \
  --set encryption.type=wireguard

Both modes encrypt all pod-to-pod traffic crossing node boundaries without application changes. WireGuard keys are automatically generated and rotated by Cilium.

Native mTLS with ztunnel (beta): recent Cilium releases can optionally use ztunnel - the same lightweight, per-node Rust proxy that powers Istio's ambient mesh - re-engineered to plug into Cilium's own control plane, with SPIRE as the certificate authority for workload identity. This is a joint effort between Isovalent and Microsoft: it brings connection-level mutual TLS into the datapath itself rather than relying solely on WireGuard/IPsec node-to-node tunnels, while keeping WireGuard/IPsec for non-TCP traffic and hitless upgrades. It's beta, with real limits at the time of writing - namespace-scoped enablement, TCP only, and no Cluster Mesh support when ztunnel encryption is on - so check the Cilium docs for which release it landed in and what's still unsupported before building on it.

Cluster Mesh

Cluster Mesh connects multiple Kubernetes clusters at the network level, allowing pods in one cluster to reach services in another using standard DNS and Kubernetes service names.

flowchart LR
    subgraph Cluster A
        PodA[Pod] --> SvcA[Service]
    end
    subgraph Cluster B
        SvcB[Global Service\nmirror in A]
        PodB[Pod]
    end
    SvcA --> SvcB --> PodB
cilium clustermesh enable --context cluster-a
cilium clustermesh enable --context cluster-b
cilium clustermesh connect --destination-context cluster-b

Annotate a service as global:

metadata:
  annotations:
    service.cilium.io/global: "true"
    service.cilium.io/shared: "true"

Global services load-balance across healthy endpoints in all connected clusters. Combined with service.cilium.io/affinity: "local", you get prefer-local-cluster behavior with automatic failover.

Since Cilium 1.16, Cluster Mesh runs with KVStoreMesh enabled by default: each cluster caches the state it learns from remote clusters in its own local kvstore instead of every agent watching every remote cluster directly. Local agents then sync only against their own cluster's cache, which cuts the number of remote watches an agent needs to hold open and improves both scalability and blast-radius isolation as the number of connected clusters grows.

Cilium as a service mesh

Cilium's CiliumEnvoyConfig and Cilium Service Mesh mode provide L7 traffic management (retries, timeouts, header manipulation, traffic splitting) via Envoy deployed as a per-node DaemonSet - not per-pod sidecars. This gives service mesh capabilities with a much lower resource footprint.

helm upgrade cilium cilium/cilium --reuse-values \
  --set ingressController.enabled=true \
  --set ingressController.loadbalancerMode=dedicated

Gateway API and Ingress

Because Cilium already runs an eBPF datapath and can deploy Envoy per-node, it can act as a full Kubernetes Ingress controller or Gateway API implementation, removing the need to run a separate ingress controller (nginx-ingress, HAProxy, etc.) alongside it.

helm upgrade cilium cilium/cilium --reuse-values \
  --set gatewayAPI.enabled=true
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: public-gateway
spec:
  gatewayClassName: cilium
  listeners:
    - name: https
      port: 443
      protocol: HTTPS
      tls:
        mode: Terminate
        certificateRefs:
          - name: my-cert-tls

The practical case for consolidating onto Cilium's ingress/Gateway API support: one less component to install, upgrade, and secure, one less set of iptables/proxy rules interacting with the CNI's own datapath, and ingress traffic gets the same Hubble observability as every other flow in the cluster. The tradeoff is ecosystem maturity - a dedicated ingress controller with a long track record may have edge-case features or a larger community knowledge base that a newer built-in controller hasn't accumulated yet, so evaluate feature parity for your specific ingress needs (WAF integrations, complex rewrite rules, etc.) before consolidating.

Bandwidth Manager and BBR

Bandwidth Manager uses eBPF traffic control (the same tc hooks used for policy enforcement) to enforce per-pod egress bandwidth limits and to enable fair queuing, avoiding the head-of-line blocking and bufferbloat that can occur when bursty pods share a node's network interface without any shaping.

metadata:
  annotations:
    kubernetes.io/egress-bandwidth: "10M"
helm upgrade cilium cilium/cilium --reuse-values \
  --set bandwidthManager.enabled=true \
  --set bandwidthManager.bbr=true

With bandwidthManager.bbr=true, Cilium enables the kernel's BBR (Bottleneck Bandwidth and RTT) congestion control algorithm for pod traffic instead of the default CUBIC. BBR models the actual bottleneck bandwidth and round-trip time rather than reacting to packet loss, which tends to perform meaningfully better on high-bandwidth, high-latency paths (cross-region traffic, paths with shallow buffers) where CUBIC's loss-based backoff leaves throughput on the table. This requires a kernel with BBR support compiled in (most modern distributions ship it).

Egress Gateway

Many enterprise environments firewall outbound traffic by source IP - a third-party API, a partner's on-prem system, or a legacy internal service will only accept connections from an allow-listed address. In a Kubernetes cluster, pod IPs are ephemeral and node IPs can be numerous, which makes a naive "allow-list the cluster" approach impractical.

Egress Gateway solves this by routing selected pods' egress traffic through one or a small set of designated gateway nodes, so external services see a small, stable set of source IPs regardless of which node a pod actually schedules on.

flowchart LR
    PodA["Pod\n(node 3)"] -.-> GW["Egress Gateway node\n(stable IP: 203.0.113.10)"]
    PodB["Pod\n(node 7)"] -.-> GW
    GW --> Ext["External service\n(firewalled by source IP)"]
apiVersion: cilium.io/v2
kind: CiliumEgressGatewayPolicy
metadata:
  name: partner-api-egress
spec:
  selectors:
    - podSelector:
        matchLabels:
          app: billing
  destinationCIDRs:
    - "198.51.100.0/24"
  egressGateway:
    nodeSelector:
      matchLabels:
        egress-gateway: "true"
    egressIP: "203.0.113.10"

Traffic from matching pods to matching destination CIDRs is SNAT'd to the configured egressIP as it leaves the cluster, regardless of which node the pod is actually running on. This is one of the more common reasons enterprises adopt Cilium specifically - it solves a real, otherwise-painful firewall-allow-listing problem without requiring a NAT gateway appliance or cloud-specific egress IP feature.

BGP control plane

Cilium can advertise pod and service CIDRs to upstream BGP routers, enabling bare-metal clusters to expose LoadBalancer services without a cloud provider. Use the BGPv2 API (CiliumBGPClusterConfig, CiliumBGPPeerConfig, CiliumBGPAdvertisement, introduced in Cilium 1.16). The older CiliumBGPPeeringPolicy is deprecated and on a removal path, so clusters still running v1 peering-policy manifests need to migrate - because this gates an upgrade, confirm the exact deprecation and removal releases in the Cilium upgrade guide for your target version rather than taking a version number from this page:

apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata:
  name: rack-bgp
spec:
  nodeSelector:
    matchLabels:
      rack: "1"
  bgpInstances:
    - name: instance-65001
      localASN: 65001
      peers:
        - name: tor-router
          peerAddress: 10.0.0.1
          peerASN: 65000
          peerConfigRef:
            name: peer-config

Troubleshooting playbook

Work through these in order when something in the datapath isn't behaving as expected - each tool narrows the problem to a smaller layer:

  1. cilium status --verbose - the first stop for any issue. Confirms the agent is healthy, kube-proxy replacement is actually active, and the control-plane connection to the Kubernetes API is up. A Cilium: Warning or Degraded status here usually points straight at the root cause.
kubectl -n kube-system exec ds/cilium -- cilium status --verbose
  1. cilium-health - tests actual node-to-node and pod-to-pod connectivity through the datapath, independent of any specific application. Useful for distinguishing "the network fabric itself is broken" from "policy is blocking this specific traffic":
kubectl -n kube-system exec ds/cilium -- cilium-health status
  1. cilium monitor - a live, low-level stream of datapath events (drops, policy verdicts, debug traces) directly from the eBPF programs, filterable by type. This is a level below Hubble - useful when Hubble itself isn't seeing what you expect, or for drops that occur before Hubble's own observation point:
kubectl -n kube-system exec ds/cilium -- cilium monitor --type drop
kubectl -n kube-system exec ds/cilium -- cilium monitor -v --related-to <endpoint-id>
  1. eBPF map pressure - every eBPF map (policy, connection tracking, NAT, load-balancing) has a fixed maximum size set at agent startup. Once a map fills, new entries fail to insert and you get silent packet drops or new connections failing while existing ones keep working - a classic "some things fail intermittently at scale" symptom. Check occupancy against configured limits before it becomes an outage:
kubectl -n kube-system exec ds/cilium -- cilium bpf policy list
kubectl -n kube-system exec ds/cilium -- cilium bpf nat list
kubectl -n kube-system exec ds/cilium -- cilium bpf ct list global | wc -l
kubectl -n kube-system exec ds/cilium -- cilium map get cilium_ct_any4_global   # shows max entries vs current

Map sizes are tunable via Helm values (e.g. bpf.ctTcpMax, bpf.natMax) - on very large or very high-connection-churn clusters, size them proactively rather than reactively once drops start.

Operational patterns

Debug policy drops: hubble observe --verdict DROPPED is far faster than reading iptables logs. Every drop includes the policy name and direction that blocked it.

Identity-based policy: Cilium assigns a numeric identity to each endpoint based on its labels. Policy evaluation uses identities, not IPs. This means policy is stable across pod restarts and IP reassignments.

Monitor eBPF map pressure: cilium bpf policy list and cilium bpf nat list show map occupancy. At very high scale, tune map sizes before running into hard limits.

Bandwidth management: Cilium supports egress bandwidth limits via eBPF traffic control, without requiring a separate rate-limiting sidecar.

Common mistakes

Leaving kube-proxy's iptables rules in place after enabling Cilium's kube-proxy replacement. Scaling the kube-proxy DaemonSet to zero doesn't remove its rules. Stale KUBE-SERVICES/KUBE-NODEPORTS chains produce intermittent, hard-to-diagnose connection failures that look like flaky infrastructure. Explicitly clean up old rules, or start from a cluster that never ran kube-proxy.

Writing CiliumNetworkPolicy egress rules against pod or node IPs instead of toEndpoints selectors or toFQDNs. IPs are ephemeral in Kubernetes; a policy pinned to an IP silently stops matching the moment a pod is rescheduled or a backend rotates. Use label-based toEndpoints for in-cluster destinations and toFQDNs for external ones.

Deriving a CiliumNetworkPolicy from too short a Hubble observation window. A policy built from five minutes of traffic will miss anything that happens on a longer cycle - nightly batch jobs, periodic health checks to a third party, cert renewal callbacks - and those dependencies will start failing silently the moment the policy goes into enforcing mode.

Ignoring maxEjectionPercent-equivalent safety limits when sizing eBPF maps. Sizing connection-tracking or NAT maps too small for actual cluster scale causes drops that look like application bugs; the fix is capacity planning based on real endpoint and connection counts, not the defaults from a much smaller test cluster.

Assuming Hubble UI's service map is a complete picture without checking the observation time range. Like the policy-generation flow above, the UI only shows what happened during the window it queried - a service that looks isolated may simply not have talked to anything during that window.

  • Istio - the sidecar/ambient mesh Cilium's service-mesh mode is usually compared against
  • Linkerd - the lightweight mesh alternative
  • Network Policies - the upstream API CiliumNetworkPolicy extends
  • Networking Overview - how the CNI fits into the wider networking model
  • Gateway API - Cilium is a conformant Gateway API implementation