OpenTelemetry¶

Who this page is for: in plain English, OpenTelemetry is one standard way for your applications to emit traces, metrics, and logs, so you can send them anywhere without rewriting instrumentation. This page is production depth, aimed at readers already comfortable with Pods, Services, and Deployments. New to Kubernetes? Start with the Beginner track.

OpenTelemetry (OTel) is the CNCF standard for generating, collecting, and exporting telemetry - traces, metrics, and logs - in a vendor-neutral format. It replaced a fragmented landscape of proprietary agents and half-compatible tracing SDKs (OpenTracing, OpenCensus merged into it) with a single specification, a single wire protocol, and one set of SDKs per language.

The problem it solves: without a standard, instrumenting an application meant picking a vendor's SDK and being locked into that vendor's backend. OpenTelemetry decouples instrumentation from the backend - you instrument once, emit data over OTLP, and route it to Prometheus, Jaeger, Tempo, Datadog, or any combination, without touching application code again.

The three signals¶

OpenTelemetry defines three telemetry types, often called the "three pillars," though OTel's own project direction increasingly treats them as correlated signals that should share context (a trace ID on a log line, exemplars linking a metric to the trace that produced it) rather than three disconnected silos. All three signals' specifications - including the OTLP data model for each - are now stable/GA, though language-specific SDK implementations vary in how completely they've caught up to spec.

For PromQL and Prometheus's own data model, see the Prometheus guide - this page focuses on how OpenTelemetry generates and routes signals, not on querying them once they land.

Architecture¶

flowchart TD
    subgraph Application Pods
        App1["App\n(auto or manual\nSDK instrumentation)"]
        App2["App\n(auto or manual\nSDK instrumentation)"]
    end
    subgraph Node Agent Tier
        AgentCollector["OTel Collector\n(DaemonSet)"]
    end
    subgraph Gateway Tier
        GatewayCollector["OTel Collector\n(Deployment)\nreceivers -> processors -> exporters"]
    end
    subgraph Backends
        Prom[Prometheus]
        Jaeger[Jaeger / Tempo]
        Logs[Loki / Elasticsearch]
    end
    App1 -- OTLP grpc/http --> AgentCollector
    App2 -- OTLP grpc/http --> AgentCollector
    AgentCollector -- OTLP --> GatewayCollector
    GatewayCollector --> Prom
    GatewayCollector --> Jaeger
    GatewayCollector --> Logs

Applications emit telemetry over OTLP to a nearby Collector; Collectors process and re-export it to one or more backends. Nothing in an application ever talks directly to Prometheus or Jaeger - the Collector is the universal integration point, which is also why OTel deprecated its own earlier standalone exporters/agents in favor of routing everything through Collector pipelines.

OTLP: the wire protocol¶

OTLP (OpenTelemetry Protocol) is the transport all SDKs and Collectors speak. It comes in two forms:

• OTLP/gRPC - default port 4317, binary protobuf over HTTP/2, lower overhead, preferred for service-to-collector and collector-to-collector traffic.
• OTLP/HTTP - default port 4318, protobuf or JSON over plain HTTP/1.1, useful when gRPC is blocked (browsers, some serverless environments, restrictive egress).

# Typical env vars an SDK reads to find its collector
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.observability:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_SERVICE_NAME=checkout
OTEL_RESOURCE_ATTRIBUTES=service.namespace=my-app,deployment.environment=production

Because OTLP is a single standard protocol, any two OTel-speaking components interoperate regardless of vendor - an SDK from one team's language and a Collector from another's deployment both just speak OTLP.

Instrumentation model¶

There are two ways telemetry gets generated, and most real deployments use both.

Manual SDK instrumentation - you import the language SDK, create spans/meters/loggers explicitly, and control exactly what's recorded. This is the only option for custom business-logic spans (e.g., "time spent in the pricing engine") and gives you full control over attributes.

Auto-instrumentation - an agent or bytecode-injection library instruments common frameworks (HTTP servers, database clients, RPC frameworks) without code changes. This gets you HTTP/DB/RPC spans and basic metrics for free, at the cost of coarser granularity than hand-written spans.

The OpenTelemetry Operator and pod-annotation injection¶

In Kubernetes, the OpenTelemetry Operator automates both Collector deployment and auto-instrumentation injection. An Instrumentation custom resource defines the auto-instrumentation config per language, and workloads opt in via a pod annotation - the injector webhook adds the appropriate init container and env vars at admission time.

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: java-instrumentation
  namespace: my-app
spec:
  exporter:
    endpoint: http://otel-collector.observability:4317
  propagators:
    - tracecontext
    - baggage
  sampler:
    type: parentbased_traceidratio
    argument: "0.1"

metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-java: "true"     # or inject-python, inject-nodejs, inject-dotnet, inject-go

Supported auto-injection languages via the Operator include Java, Python, Node.js, .NET, and Go (Go instrumentation uses eBPF-based injection rather than bytecode manipulation, since Go has no runtime to hook into the way a JVM or Node process does).

The OpenTelemetry Collector¶

The Collector is the centerpiece of most real deployments: a standalone, vendor-agnostic service that receives telemetry, transforms it, and exports it - all configured declaratively as a pipeline.

flowchart LR
    subgraph Receivers
        R1[otlp]
        R2[prometheus]
    end
    subgraph Processors
        P1[memory_limiter]
        P2[batch]
        P3[resource / k8sattributes]
        P4[tail_sampling]
    end
    subgraph Exporters
        E1[otlp -> gateway]
        E2[prometheusexporter]
        E3[otlp -> Jaeger/Tempo]
    end
    R1 --> P1 --> P3 --> P4 --> P2 --> E1
    R1 --> P1 --> P3 --> P2 --> E3
    R2 --> P1 --> P2 --> E2

Receivers ingest data - otlp (gRPC/HTTP), prometheus (scrape-compatible), filelog, jaeger, zipkin, and many others.

Processors transform data in-flight - batch groups records for efficient export, memory_limiter prevents OOM under load spikes, resource/k8sattributes enriches spans and metrics with Kubernetes metadata (k8s.pod.name, k8s.namespace.name), tail_sampling makes sampling decisions after seeing a full trace. Note that tail_sampling, k8sattributes, and the loadbalancing exporter mentioned below all ship in the Collector Contrib distribution, not Core - if you're building a custom Collector image or picking an Operator OpenTelemetryCollector image, make sure it's the contrib (or an equivalent vendor) build, not the minimal core one.

Exporters send data onward - otlp (to another Collector or OTel-native backend), prometheusexporter (exposes a /metrics endpoint Prometheus can scrape, or prometheusremotewriteexporter to push directly), otlp/vendor-specific exporters toward Jaeger, Tempo, or a commercial backend.

Pipelines wire receivers, processors, and exporters together per signal type:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
  k8sattributes:
    extract:
      metadata:
        - k8s.pod.name
        - k8s.namespace.name
        - k8s.deployment.name
  batch:
    send_batch_size: 8192
    timeout: 5s
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: sample-errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: sample-slow
        type: latency
        latency:
          threshold_ms: 500
      - name: baseline-rate
        type: probabilistic
        probabilistic:
          sampling_percentage: 10

exporters:
  otlp/tempo:
    endpoint: tempo.observability:4317
    tls:
      insecure: true
  prometheusremotewrite:
    endpoint: http://prometheus.observability:9090/api/v1/write

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, tail_sampling, batch]
      exporters: [otlp/tempo]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, batch]
      exporters: [prometheusremotewrite]

Deployed via the Operator, this becomes an OpenTelemetryCollector custom resource rather than a raw Deployment/ConfigMap pair:

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: gateway
  namespace: observability
spec:
  mode: deployment
  replicas: 3
  config:
    receivers:
      otlp:
        protocols:
          grpc: {}
          http: {}
    exporters:
      otlp/tempo:
        endpoint: tempo.observability:4317
        tls:
          insecure: true
    service:
      pipelines:
        traces:
          receivers: [otlp]
          exporters: [otlp/tempo]

Deployment patterns¶

Two Collector deployment modes cover most needs, and larger clusters typically run both together.

The common agent + gateway topology: applications send OTLP to a local DaemonSet agent (fast, low-latency, adds k8s resource attributes), which forwards to a centralized gateway Deployment that does tail sampling and exports to backends. This keeps expensive operations like tail sampling off the node-local hot path and concentrated where a full view of each trace is available.

Context propagation¶

A trace spans multiple services by propagating a trace context in request headers. OpenTelemetry uses the W3C Trace Context standard by default:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
tracestate: vendor1=value1

traceparent encodes the trace ID, the parent span ID, and trace flags (including the sampling decision). Every instrumented HTTP client/server pair reads the incoming header, creates a child span, and forwards the header on outbound calls - this is what makes a trace a connected graph across service boundaries rather than a set of disjoint spans. baggage is a related but separate header for propagating arbitrary key-value context (not telemetry data itself) alongside a request.

Unlike a service mesh's sidecar, which can inject/forward some headers transparently, trace context propagation still requires instrumentation cooperation inside the process - auto-instrumentation libraries handle this automatically for supported frameworks, but a custom transport (e.g., a hand-rolled message queue client) needs explicit propagation code.

Semantic conventions¶

Semantic conventions standardize attribute names and values across languages and vendors, so a dashboard or query built against k8s.pod.name or http.method works regardless of which SDK or Collector produced the data.

Without shared conventions, every vendor's agent invents its own attribute names, and cross-vendor tooling (a Grafana dashboard built for one backend, reused against another) silently breaks. The k8sattributes processor in the Collector is what actually populates the Kubernetes-namespace attributes from the Kubernetes API and Downward API, based on the pod IP or resource metadata already on the incoming telemetry.

Semantic conventions do get renamed and re-stabilized over time as they mature - db.system was renamed to db.system.name (with many enum values updated to a <vendor>.<product> pattern) as part of stabilizing the database conventions, and deployment.environment similarly became deployment.environment.name. Instrumentation libraries generally ship an opt-in env var (OTEL_SEMCONV_STABILITY_OPT_IN) to control whether they emit the old attribute names, the new stable ones, or both during a migration window - worth checking before assuming a dashboard built against an older attribute name still matches incoming data after an SDK upgrade.

Sampling strategies¶

Recording every single span is often too expensive at scale. OpenTelemetry supports sampling at two different points, with different trade-offs.

Head-based sampling - the decision to sample a trace is made at the start, before any spans exist, typically as a fixed probability (e.g., parentbased_traceidratio at 10%). Cheap and simple, but it can't account for what happens later in the trace - an important error deep in the call chain may get dropped along with everything else in that 90%.

Tail-based sampling - the decision is deferred until the full trace (or a time window of it) has been collected, so policies can act on final trace properties: always keep traces containing an error, always keep traces above a latency threshold, and take a low-probability sample of everything else. This requires the Collector's tail_sampling processor and, critically, requires all spans of a given trace to arrive at the same Collector instance to make a coherent decision - which is exactly why tail sampling belongs in the gateway tier, not the per-node agent tier, and typically needs a load balancer that routes by trace ID (the loadbalancing exporter) when running multiple gateway replicas.

Common mistakes¶

• Running tail sampling on a horizontally scaled Collector without trace-ID-aware routing. If spans of one trace land on different gateway replicas, no single replica ever sees the whole trace and the sampling policy can't work correctly. Use the loadbalancing exporter to route by trace ID upstream of tail sampling.
• Skipping memory_limiter. A Collector under a telemetry burst with no memory limiter configured can OOM and crash-loop, taking down the pipeline exactly when you need it most (during an incident, which is when trace volume spikes).
• Sending raw OTLP straight to a backend from every pod. Bypassing the Collector loses batching, retry buffering, and centralized enrichment/redaction, and couples every application directly to backend availability and API shape.
• Treating auto-instrumentation as sufficient for everything. It covers framework boundaries well but won't produce a span for "time spent validating this order" - business-logic-level visibility still needs manual spans.
• Ignoring semantic conventions in custom attributes. Inventing your own pod_name attribute instead of using k8s.pod.name breaks cross-tool correlation and defeats the point of a standard.
• Forgetting that tracestate/traceparent must be forwarded by custom transports. Message queues, batch jobs, and non-HTTP RPC often need explicit propagation code; assuming it "just works" like it does for instrumented HTTP frequently produces broken, disconnected traces.

Source Links¶

• OpenTelemetry documentation
• OpenTelemetry specification
• OTLP protocol specification
• Semantic conventions (including k8s.* resource attributes)
• OpenTelemetry Collector documentation
• Collector deployment patterns (agent vs gateway)
• opentelemetry-collector-contrib (tail_sampling, loadbalancing, k8sattributes)
• OpenTelemetry Operator
• Zero-code / auto-instrumentation
• Context propagation and W3C Trace Context
• OpenTelemetry project status and releases

Related Concepts¶

• Jaeger - the tracing backend most often paired with an OTel Collector
• Prometheus - the usual destination for OTel metrics, via the Prometheus remote-write or OTLP endpoint
• Istio - mesh-generated spans that need the same propagation headers
• Cilium - Hubble flow data as a complement to application-level traces
• Troubleshooting - what to do when the traces don't arrive