Skip to content

Istio

Who this page is for: in plain English, Istio is a service mesh - it inserts proxies around your workloads so traffic between services is encrypted, routable, and observable without touching application code. This page is production depth, aimed at readers already comfortable with Pods, Services, and Deployments. New to Kubernetes? Start with the Beginner track.

Istio is a service mesh that adds traffic management, mutual TLS, and observability to Kubernetes workloads without requiring application code changes.

The core problem it solves: as you add more services, cross-service concerns like retries, circuit breaking, encryption, and access control accumulate in each service. A mesh moves those concerns to the infrastructure layer.

A note on versions and distributions: Istio is a CNCF graduated project (it graduated in 2023, having been donated to the CNCF the year before) and releases on a roughly quarterly cadence, with each minor release supported for a limited window - see the Istio release and support policy for which minors are still receiving patches, because upgrades are not optional for long on this project. Stewardship is multi-vendor (Google, IBM, Solo.io, Tetrate, Red Hat and others), and commercial distributions exist - Solo.io's Gloo Mesh, Tetrate Istio Distro, Red Hat OpenShift Service Mesh - built on the same upstream. Two structural choices on this page version independently of Istio itself: the ambient data plane, and whether you drive routing through Istio's own VirtualService/Gateway APIs or through Gateway API, which Istio also implements.

Architecture

Istio separates into a control plane and a data plane.

flowchart TD
    subgraph Control Plane
        istiod["istiod\n(Pilot + Citadel + Galley)"]
    end
    subgraph Pod A
        AppA[App container] <--> ProxyA[Envoy sidecar]
    end
    subgraph Pod B
        AppB[App container] <--> ProxyB[Envoy sidecar]
    end
    istiod -- xDS config --> ProxyA
    istiod -- xDS config --> ProxyB
    istiod -- certificates --> ProxyA
    istiod -- certificates --> ProxyB
    ProxyA <-- mTLS --> ProxyB

istiod is a single binary that consolidates three former components: - Pilot: distributes routing rules and service discovery to Envoy proxies via xDS APIs - Citadel: issues and rotates mTLS certificates (SPIFFE/X.509 format) - Galley: validates and processes configuration

Envoy sidecars intercept all inbound and outbound traffic from every pod. The application has no awareness of them - iptables rules redirect traffic through the sidecar.

Sidecar injection

Namespaces with istio-injection: enabled label get automatic injection:

kubectl label namespace my-app istio-injection=enabled

Or opt individual pods in/out:

metadata:
  annotations:
    sidecar.istio.io/inject: "true"   # or "false"

Traffic management

Istio traffic management uses two primary CRDs: VirtualService and DestinationRule.

VirtualService

A VirtualService defines how requests to a host are routed. It replaces the coarse-grained behavior of a Kubernetes Service with fine-grained routing rules.

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: checkout
spec:
  hosts:
    - checkout
  http:
    - match:
        - headers:
            x-canary:
              exact: "true"
      route:
        - destination:
            host: checkout
            subset: canary
    - route:
        - destination:
            host: checkout
            subset: stable
          weight: 100

DestinationRule

A DestinationRule defines subsets and per-subset policies (load balancing, connection pool, outlier detection):

apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: checkout
spec:
  host: checkout
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        http1MaxPendingRequests: 50
        maxRequestsPerConnection: 10
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 30s
      baseEjectionTime: 30s
  subsets:
    - name: stable
      labels:
        version: stable
    - name: canary
      labels:
        version: canary

outlierDetection is passive circuit breaking - Istio ejects hosts that return errors above the threshold. For active circuit breaking (fail fast when the pool is exhausted), configure connectionPool.

Retries and timeouts

http:
  - route:
      - destination:
          host: payment
    timeout: 3s
    retries:
      attempts: 3
      perTryTimeout: 1s
      retryOn: gateway-error,connect-failure,retriable-4xx

Set timeouts and retries at the mesh layer, not in application code. This prevents timeout stacking - if service A calls B which calls C, and all have 5s timeouts, the end-to-end latency can reach 15s.

Ingress Gateway

Istio's Gateway resource exposes services outside the cluster via the istio-ingressgateway pods:

apiVersion: networking.istio.io/v1
kind: Gateway
metadata:
  name: public-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 443
        name: https
        protocol: HTTPS
      tls:
        mode: SIMPLE
        credentialName: my-cert-tls
      hosts:
        - api.example.com
---
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: api-vs
spec:
  hosts:
    - api.example.com
  gateways:
    - public-gateway
  http:
    - route:
        - destination:
            host: api-service
            port:
              number: 8080

Mutual TLS

Istio issues SPIFFE-compliant X.509 certificates to every sidecar and rotates them automatically (default 24-hour lifetime). mTLS validates both sides of every service-to-service connection.

PeerAuthentication

Controls whether mTLS is enforced or optional for inbound traffic:

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: my-app
spec:
  mtls:
    mode: STRICT   # reject any plaintext connections

Modes: - STRICT: all inbound connections must use mTLS - PERMISSIVE: accept both mTLS and plaintext (useful during migration) - DISABLE: plaintext only

Apply STRICT at the mesh level (namespace: istio-system) to lock down the entire cluster, then use PERMISSIVE per namespace for services that receive external traffic.

Authorization policies

AuthorizationPolicy controls which services or users can reach a workload:

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: allow-frontend-to-checkout
  namespace: my-app
spec:
  selector:
    matchLabels:
      app: checkout
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              - cluster.local/ns/my-app/sa/frontend
      to:
        - operation:
            methods: ["POST"]
            paths: ["/checkout/*"]

principals maps to the SPIFFE ID embedded in the mTLS certificate - cluster.local/ns/<namespace>/sa/<service-account>. This is identity-based authorization tied to real Kubernetes service accounts, not IP addresses.

Default-deny pattern: apply an empty AuthorizationPolicy to a namespace - it denies everything, then add explicit ALLOW policies per service.

Observability

Istio automatically generates three telemetry signals without application instrumentation:

Metrics - Envoy exports Prometheus metrics for every request: istio_requests_total, istio_request_duration_milliseconds, istio_request_bytes. These are available at :15090/metrics on every sidecar.

Distributed traces - Istio propagates trace headers (B3 or W3C TraceContext). Applications must forward the headers (x-request-id, x-b3-traceid, etc.) between service calls - Envoy adds them on ingress but can't forward them internally without application cooperation.

Access logs - Envoy logs every request/response. Format is configurable via Telemetry CRD.

apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  accessLogging:
    - providers:
        - name: envoy
  tracing:
    - providers:
        - name: zipkin
      randomSamplingPercentage: 1.0

Ambient mesh (sidecarless)

Istio's ambient mode, which reached General Availability in Istio 1.24 (November 2024), removes sidecars entirely. It splits the data plane into two layers instead of bundling L4 and L7 concerns into a single per-pod proxy.

flowchart TD
    subgraph Node
        PodA[App pod A] --> ztunnel[ztunnel\nper-node DaemonSet]
        PodB[App pod B] --> ztunnel
    end
    ztunnel -- "L4 mTLS tunnel\nHBONE (HTTP/2 CONNECT)" --> ztunnel2[ztunnel\non remote node]
    subgraph "Namespace with waypoint"
        waypoint[waypoint proxy\nEnvoy, per-ServiceAccount]
    end
    ztunnel2 -. "L7 traffic routed through\nwaypoint when policy requires it" .-> waypoint
    waypoint --> PodC[App pod C]

ztunnel is a purpose-built, minimal proxy (written in Rust, not Envoy) that runs one instance per node as a DaemonSet. It handles only L4: it establishes mTLS tunnels between nodes using HBONE (an HTTP/2 CONNECT-based tunneling protocol), carries the workload identity (SPIFFE) for every connection, and enforces L4 authorization policy (allow/deny by identity, namespace, or IP). It does not parse HTTP, does not do retries, and does not do per-path routing. Because it's not proxying every pod's traffic through a full L7 proxy, per-pod memory and CPU overhead essentially disappears - workloads that previously each carried an Envoy sidecar now share one lightweight per-node process.

waypoint proxies are the L7 layer, and they're just Envoy - the same proxy sidecar mode uses, but deployed as a separate workload (a Deployment, not injected into every pod) and only for the identities that need L7 features. When a namespace or service account has a waypoint, ztunnel redirects matching traffic to it for HTTP-aware routing, retries, timeouts, circuit breaking, and L7 AuthorizationPolicy rules (path/method matching). Traffic that doesn't need L7 policy skips the waypoint entirely and stays on the cheaper ztunnel-only path.

The operational tradeoff: ambient mode gets you mesh-wide mTLS and L4 identity-based policy essentially for free, with dramatically lower baseline resource cost and no per-pod injection to manage. What you give up, unless you deploy a waypoint, is per-pod L7 granularity - HTTP retries, header-based routing, and L7 authorization all require a waypoint in the path, which reintroduces an Envoy hop (and its resource cost) for that traffic. In practice, a common pattern is: run the whole mesh in ambient mode for baseline mTLS and L4 policy, and add waypoints selectively only to the namespaces or services that actually need L7 traffic management - rather than paying the L7 proxy tax everywhere the sidecar model forces on you by default.

istioctl install --set profile=ambient
kubectl label namespace my-app istio.io/dataplane-mode=ambient

Deploy a waypoint for L7 policies, retries, and traffic splitting on a namespace:

istioctl waypoint apply --namespace my-app --enroll-namespace

Sidecar and ambient modes can coexist in the same mesh during migration - label namespaces individually and move workloads over incrementally. A workload in ambient mode still gets a VirtualService/DestinationRule applied identically to sidecar mode; the only difference is which proxy enforces it.

Multi-cluster mesh topologies

Istio supports several ways to extend a mesh across cluster boundaries, all of which rely on an east-west gateway - a dedicated ingress gateway that terminates mTLS from remote clusters and forwards to local sidecars, rather than exposing the mesh's internal mTLS ports directly to the internet.

Primary-remote: one cluster runs the full control plane (istiod); remote clusters run only the data plane and read configuration from the primary over the network. Simpler to operate (one control plane to upgrade and reason about) but the remote clusters have a hard dependency on the primary's availability.

flowchart LR
    subgraph "Cluster A (primary)"
        istiod[istiod]
        GWA[east-west gateway]
    end
    subgraph "Cluster B (remote)"
        GWB[east-west gateway]
        ProxyB[Envoy sidecars]
    end
    istiod -- xDS over network --> ProxyB
    GWA <-- mTLS --> GWB

Multi-primary: every cluster runs its own istiod, and each control plane discovers services in every other cluster via the Kubernetes API (each cluster needs a remote-secret granting API access to the others). No single point of control-plane failure, at the cost of running and upgrading N control planes in lockstep.

Both topologies rely on the same mechanism for cross-cluster service discovery: a Service with the same name and namespace in two clusters is treated as one logical service, and Istio load-balances across all healthy endpoints in every connected cluster, using the east-west gateway to route between clusters and locality-weighted load balancing to prefer same-cluster/same-zone endpoints when they're healthy.

# Generate a remote secret so cluster A's istiod can discover cluster B's endpoints
istioctl create-remote-secret --context=cluster-b --name=cluster-b | \
  kubectl apply -f - --context=cluster-a

Circuit breaking in practice

The outlierDetection and connectionPool fields on a DestinationRule work together but solve different problems: connectionPool limits how much concurrent load a client sends to a destination (proactive), while outlierDetection ejects specific unhealthy endpoints from the load-balancing pool based on observed behavior (reactive).

apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: payment
spec:
  host: payment
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 50            # cap concurrent TCP connections per client sidecar
      http:
        http1MaxPendingRequests: 20   # requests queued waiting for a free connection
        maxRequestsPerConnection: 10  # recycle connections to avoid pinning to one backend
    outlierDetection:
      consecutive5xxErrors: 5         # eject after 5 consecutive 5xx from one endpoint
      interval: 10s                   # how often to scan for outliers
      baseEjectionTime: 30s           # minimum ejection duration
      maxEjectionPercent: 50          # never eject more than half the pool at once

Two details that catch people out: ejection is per-client-sidecar, not global - every calling sidecar makes its own ejection decision based on what it individually observes, so different clients can have different views of which endpoints are "up." And maxEjectionPercent is a safety valve - without it, a bad deploy that makes every replica unhealthy can eject the entire pool and turn a partial outage into a total one; capping ejection at 50% guarantees some capacity stays in rotation even during a systemic failure, trading complete correctness for availability.

Traffic mirroring

Mirroring (shadow traffic) sends a copy of live production traffic to a new version without returning its response to the caller - the mirrored request's response is discarded. This validates a new version under real traffic and load patterns before it serves a single real user.

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: checkout
spec:
  hosts:
    - checkout
  http:
    - route:
        - destination:
            host: checkout
            subset: stable
          weight: 100
      mirror:
        host: checkout
        subset: canary
      mirrorPercentage:
        value: 10.0   # mirror 10% of production traffic to canary

Mirrored requests get a -shadow suffix appended to their Host/Authority header, which is useful for distinguishing shadow traffic in the canary's logs and metrics. Mirroring is one-way and fire-and-forget by design - never mirror traffic to a version that performs writes (payments, order creation) unless the mirrored path is explicitly idempotent or backed by a sandboxed dependency, since the canary will actually execute the request, just not answer the original caller.

Debugging the mesh

istioctl analyze runs a static set of checks against live cluster config and flags common misconfigurations before they cause outages - dangling VirtualService host references, conflicting Gateway port bindings, AuthorizationPolicy selectors that match nothing, missing PeerAuthentication conflicts. Run it before and after every config change:

istioctl analyze -n my-app
istioctl analyze --all-namespaces

istioctl proxy-config dumps the actual xDS config a running Envoy sidecar has received from istiod - this is the ground truth for "what is this proxy actually doing," independent of what you think you applied:

istioctl proxy-config clusters <pod> -n my-app     # upstream clusters (destinations)
istioctl proxy-config listeners <pod> -n my-app    # inbound/outbound listeners and filter chains
istioctl proxy-config routes <pod> -n my-app       # route tables referenced by listeners
istioctl proxy-config endpoints <pod> -n my-app    # individual backend IPs and their health status
istioctl proxy-config secret <pod> -n my-app       # certificates the proxy has loaded

A mismatch between what you expect (based on your VirtualService/DestinationRule YAML) and what proxy-config actually shows is the single most common source of "my routing rule isn't working" issues - usually caused by a typo'd host, a DestinationRule subset that doesn't match any pod labels, or config that was pushed to one revision of istiod but not another during a canary control-plane upgrade.

Diagnosing mTLS handshake failures: the most common symptom is upstream connect error or disconnect/reset before headers or 503 UF in access logs. Work through it in order:

  1. Check both sides agree on mTLS mode - istioctl proxy-config secret <pod> to confirm the client has a valid, non-expired cert; a PeerAuthentication set to STRICT on the server side while a caller has no sidecar (and so sends plaintext) is a frequent cause.
  2. Check for a DestinationRule accidentally forcing tls.mode: DISABLE or ISTIO_MUTUAL mismatched against the server's PeerAuthentication mode - this is a common source of confusing failures because the error looks like a network problem, not a config problem.
  3. Use istioctl proxy-config listener <pod> --port <port> -o json on the server side to confirm a sidecar is actually injected and listening - traffic to an uninjected pod fails mTLS by definition since there's no sidecar terminating it.
  4. openssl s_client against the sidecar's mTLS port from inside another pod, or tcpdump inside the pod network namespace, can confirm whether a TLS handshake is even being attempted versus failing at the TCP layer (e.g. a NetworkPolicy blocking the connection before Istio ever sees it).

Gateway API support

Istio implements the Kubernetes-standard Gateway API as its recommended ingress mechanism going forward, in preference to the Istio-specific Gateway/VirtualService CRDs shown earlier in this page. The vendor-neutral resources (Gateway, HTTPRoute, TCPRoute) are portable across implementations (Istio, Cilium, Envoy Gateway, cloud LB controllers) and are actively where new routing features land first.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: public-gateway
spec:
  gatewayClassName: istio
  listeners:
    - name: https
      port: 443
      protocol: HTTPS
      tls:
        mode: Terminate
        certificateRefs:
          - name: my-cert-tls
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-route
spec:
  parentRefs:
    - name: public-gateway
  hostnames:
    - "api.example.com"
  rules:
    - backendRefs:
        - name: api-service
          port: 8080

Istio's VirtualService/Gateway CRDs aren't deprecated and remain fully supported, but new deployments should default to Gateway API - it's what waypoint proxies use natively for L7 policy in ambient mode, and Istio's own roadmap treats it as the primary interface. Existing VirtualService-based configs can be migrated incrementally since both mechanisms can coexist in the same mesh.

Sidecar resource scoping

In large meshes (hundreds of services), every sidecar receives the full mesh-wide xDS config by default - every proxy learns about every service, even ones it will never call. This bloats config push size, slows proxy startup, and increases memory per sidecar as the mesh grows, independent of how much traffic that particular workload actually handles.

The Sidecar CRD scopes what config a proxy (or set of proxies, via a selector) actually receives, cutting both the config size pushed and the memory footprint of the resulting Envoy:

apiVersion: networking.istio.io/v1
kind: Sidecar
metadata:
  name: default
  namespace: my-app
spec:
  egress:
    - hosts:
        - "./*"              # services in this namespace
        - "istio-system/*"   # the mesh's own control-plane/gateway services
        - "shared-svcs/checkout.shared-svcs.svc.cluster.local"

A Sidecar resource named default with no workloadSelector applies to every workload in its namespace and is the standard way to scope an entire namespace down to only the hosts it legitimately talks to. For meshes with hundreds of services, applying default Sidecar scoping per namespace is one of the highest-leverage changes for both config-push latency and steady-state sidecar memory.

Operational patterns

Canary upgrades: always upgrade with istioctl upgrade, not helm upgrade directly. Use istioctl analyze before and after to catch configuration issues. Canary the control plane itself with revision tags (istioctl install --set revision=1-24-0) so new sidecars pick up the new istiod while existing sidecars keep talking to the old one until you explicitly migrate namespaces.

Debug a sidecar: istioctl proxy-config cluster <pod>, istioctl proxy-config listener <pod>. These show what Pilot has pushed to Envoy. Mismatched config between what you expect and what Envoy has is the most common Istio issue.

Resource overhead: each Envoy sidecar uses roughly 50-100MB memory and a small but real CPU budget. For high-QPS services, tune concurrency to pin sidecar worker threads. At mesh scale, this overhead multiplies by every pod replica, which is a large part of ambient mode's appeal for cost-sensitive clusters.

Egress control: use ServiceEntry to allow workloads to reach external services, and an EgressGateway to route and log all outbound traffic centrally.

Common mistakes

Applying PeerAuthentication: STRICT mesh-wide before every workload has a sidecar (or ztunnel, in ambient mode) injected. Any uninjected pod immediately loses the ability to receive traffic, since it can't complete mTLS. Roll out with PERMISSIVE first, confirm via istioctl proxy-config secret and Hubble/Kiali-style traffic graphs that everything is actually meshed, then flip to STRICT.

Setting timeouts and retries in both the application and the mesh, uncoordinated. If a service's HTTP client has its own 10s timeout and the VirtualService also sets a 3s timeout with 3 retries, you get compounding, hard-to-predict behavior under load. Pick one layer - the mesh, ideally - and remove redundant application-level retry logic.

Forgetting that DestinationRule subsets require matching pod labels. A subset defined against version: canary silently matches zero pods, and zero-weight-effective traffic routing failures, if there's no corresponding label on any pod - istioctl proxy-config endpoints will show an empty endpoint list for that subset, which is the fastest way to confirm it.

Treating traffic mirroring as safe for all traffic. Mirrored requests are real requests that really execute - mirroring write traffic to a version under test can double-write, double-charge, or double-send emails. Mirror read paths, or make sure the mirrored destination is sandboxed.

Not scoping Sidecar resources in large meshes and being surprised by slow proxy startup and config push latency. The default all-to-all config distribution works fine for a few dozen services and becomes an operational liability past a few hundred.