Prometheus¶

Who this page is for: in plain English, Prometheus periodically fetches numbers from your services, stores them over time, and alerts when they look wrong. This page is production depth, aimed at readers already comfortable with Pods, Services, and Deployments. New to Kubernetes? Start with the Beginner track.

Prometheus is the default monitoring system for Kubernetes. It scrapes metrics from targets, stores them as time series, evaluates alerting rules, and sends notifications via Alertmanager.

A note on versions and governance: Prometheus is a CNCF graduated project - the second project to join the foundation after Kubernetes itself - with a multi-vendor maintainer group and no open-core split; the commercial ecosystem around it (Grafana Cloud, Chronosphere, Amazon Managed Prometheus, Google Managed Prometheus) builds on the same upstream and the same PromQL. The 3.x line is current as of 2026, and Prometheus 3.0 carried a small number of breaking changes from 2.x (notably around the UI, remote-write defaults, and some flag removals), so read the migration guide before upgrading a long-lived 2.x server. The kube-prometheus-stack Helm chart and the Prometheus Operator version independently of the Prometheus server itself.

The Kubernetes ecosystem is built around Prometheus. kube-state-metrics, node-exporter, kubelet, etcd, CoreDNS, and nearly every major CNCF project expose Prometheus-compatible metrics. This means your monitoring stack is mostly wiring existing endpoints together rather than building instrumentation from scratch.

Architecture¶

flowchart TD
    subgraph Data Sources
        App[Application\n/metrics endpoint]
        KSM[kube-state-metrics]
        NE[node-exporter]
        Kubelet[kubelet\ncadvisor metrics]
    end
    subgraph Prometheus
        Scraper[Scrape engine\npull-based]
        TSDB[(Local TSDB\n15d default)]
        Rules[Rule evaluator\nrecording + alerting]
    end
    subgraph Alerting
        AM[Alertmanager\nrouting + dedup]
        Slack[Slack]
        PD[PagerDuty]
        Email[Email]
    end
    subgraph Long-term
        Thanos[Thanos / Cortex\n/ VictoriaMetrics]
    end

    App --> Scraper
    KSM --> Scraper
    NE --> Scraper
    Kubelet --> Scraper
    Scraper --> TSDB
    TSDB --> Rules
    Rules --> AM
    AM --> Slack
    AM --> PD
    AM --> Email
    TSDB --> Thanos

Prometheus is pull-based - it fetches metrics from targets on a schedule. This is different from push-based systems. The implication: Prometheus needs network access to every target, and targets don't need to know where Prometheus lives.

Data model¶

Every metric is a time series identified by a name and a set of key-value labels:

http_requests_total{method="POST", status="200", handler="/api/users"} 1234 @timestamp

The four metric types:

Type	Behavior	Use for
Counter	monotonically increasing	requests, errors, bytes transferred
Gauge	can go up or down	current connections, memory, queue depth
Histogram	samples observations into buckets	request duration, response size
Summary	pre-calculated quantiles client-side	latency percentiles (less flexible than histograms)

Prefer histograms over summaries for latency. Histogram quantiles are calculated at query time from raw bucket data, so you can change the quantile you care about after collection. Summary quantiles are fixed at instrumentation time.

Native histograms: classic histograms require you to pre-define bucket boundaries, which forces a tradeoff between resolution and cardinality (each bucket is its own time series). Native histograms use a single time series with an internal exponential bucket schema computed at a chosen resolution - you get much finer resolution without the per-bucket series explosion. They're opt-in on both sides: a client library has to emit them, and the Prometheus server has to be started with the native-histogram feature enabled (--enable-feature=native-histograms, plus a scrape protocol that can carry them). The feature has been maturing across the 3.x line rather than flipping on in a single release, so check the feature flags page and the release notes for the exact server version you run before planning a migration off classic histograms.

PromQL¶

PromQL is a functional query language for selecting and aggregating time series.

Instant and range vectors¶

# instant vector -- current value of all time series matching the selector
http_requests_total{status="200"}

# range vector -- all samples in the last 5 minutes
http_requests_total{status="200"}[5m]

Rate and increase¶

Always use rate() on counters, not raw values. Counters reset on restart; rate() handles resets correctly.

# per-second rate of HTTP requests over last 5 minutes
rate(http_requests_total[5m])

# total requests in the last hour (useful for SLO burn rate)
increase(http_requests_total[1h])

Aggregation¶

# total request rate across all pods in the production namespace
sum(rate(http_requests_total{namespace="production"}[5m]))

# request rate per handler, across all pods
sum by (handler) (rate(http_requests_total{namespace="production"}[5m]))

# 99th percentile latency
histogram_quantile(0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket{job="api"}[5m]))
)

Essential Kubernetes queries¶

# CPU usage per pod (% of limit)
sum(rate(container_cpu_usage_seconds_total{container!=""}[5m])) by (pod, namespace)
  / sum(kube_pod_container_resource_limits{resource="cpu"}) by (pod, namespace)

# Memory usage vs request
container_memory_working_set_bytes{container!=""}
  / kube_pod_container_resource_requests{resource="memory"}

# Pod restart rate (last 1h)
increase(kube_pod_container_status_restarts_total[1h]) > 0

# OOMKilled pods in last 24h
kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1

# Nodes not ready
kube_node_status_condition{condition="Ready", status="true"} == 0

# Deployment rollout progress
kube_deployment_status_replicas_available / kube_deployment_spec_replicas

Prometheus Operator¶

The Prometheus Operator is the standard way to run Prometheus in Kubernetes. It introduces CRDs that let you manage Prometheus, Alertmanager, and their configuration as Kubernetes objects.

flowchart LR
    PO[Prometheus Operator] --> |watches| SM[ServiceMonitor]
    PO --> |watches| PM[PodMonitor]
    PO --> |watches| PR[PrometheusRule]
    PO --> |generates config| Prom[Prometheus]
    SM --> |scrape target| Svc[Service]
    PM --> |scrape target| Pod[Pod]
    PR --> |loaded as| Alert[Alerting rules]

ServiceMonitor¶

Tells Prometheus which services to scrape:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: api-server
  namespace: monitoring
  labels:
    team: platform          # must match Prometheus.spec.serviceMonitorSelector
spec:
  namespaceSelector:
    matchNames:
      - production
  selector:
    matchLabels:
      app: api              # matches Service labels
  endpoints:
    - port: metrics
      interval: 15s
      path: /metrics
      relabelings:
        - sourceLabels: [__meta_kubernetes_pod_node_name]
          targetLabel: node

PodMonitor¶

Scrapes pods directly, without requiring a Service:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: batch-jobs
  namespace: monitoring
spec:
  namespaceSelector:
    any: true
  selector:
    matchLabels:
      monitoring: "true"
  podMetricsEndpoints:
    - port: metrics
      interval: 30s

PrometheusRule¶

Define alerting and recording rules as a Kubernetes resource:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: api-alerts
  namespace: monitoring
  labels:
    team: platform
spec:
  groups:
    - name: api.rules
      interval: 1m
      rules:
        - record: job:http_requests:rate5m
          expr: sum(rate(http_requests_total[5m])) by (job)

        - alert: HighErrorRate
          expr: |
            sum(rate(http_requests_total{status=~"5.."}[5m])) by (job)
            / sum(rate(http_requests_total[5m])) by (job) > 0.05
          for: 5m
          labels:
            severity: critical
            team: platform
          annotations:
            summary: "High error rate on {{ $labels.job }}"
            description: "Error rate is {{ $value | humanizePercentage }} for job {{ $labels.job }}"

Recording rules¶

Recording rules pre-compute expensive queries and store the result as a new time series. Use them for: - Queries used in dashboards (run once, read many times) - High-cardinality aggregations referenced in alerts - Multi-step alert expressions

rules:
  - record: namespace:container_cpu_usage:rate5m
    expr: |
      sum by (namespace) (
        rate(container_cpu_usage_seconds_total{container!=""}[5m])
      )

Name recording rules with the convention level:metric:operations - it makes the hierarchy obvious.

Alertmanager¶

Alertmanager handles routing, deduplication, inhibition, and silencing of alerts from Prometheus.

Routing tree¶

route:
  group_by: ["alertname", "namespace"]
  group_wait: 30s       # wait before sending the first notification
  group_interval: 5m    # wait before sending an update for an existing group
  repeat_interval: 4h   # wait before re-notifying for an alert that is still firing
  receiver: slack-platform

  routes:
    - matchers:
        - severity = "critical"
      receiver: pagerduty-oncall
      continue: false

    - matchers:
        - namespace =~ "finance-.*"
      receiver: slack-finance

(The older match/match_re fields still work but are deprecated; matchers is the current syntax.)

Inhibition¶

Suppress lower-severity alerts when a higher-severity alert is firing for the same target:

inhibit_rules:
  - source_match:
      severity: critical
    target_match:
      severity: warning
    equal: ["namespace", "job"]

This prevents alert floods when a service is completely down - you get one critical alert, not ten warnings about symptoms.

Silencing¶

Silence alerts during planned maintenance:

amtool silence add alertname="HighErrorRate" namespace="production" \
  --duration 2h \
  --comment "Planned maintenance window"
amtool silence query
amtool silence expire <id>

kube-prometheus-stack¶

The kube-prometheus-stack Helm chart is the standard way to deploy the full monitoring stack:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  -f values.yaml

It bundles Prometheus Operator, Prometheus, Alertmanager, Grafana, kube-state-metrics, node-exporter, and a set of default recording rules and dashboards.

Long-term storage¶

Prometheus's local TSDB has a default 15-day retention window. For longer retention and multi-cluster federation:

Thanos - queries multiple Prometheus instances and stores data in object storage (S3, GCS, Azure Blob). The sidecar mode attaches to each Prometheus and uploads completed blocks.

VictoriaMetrics - drop-in Prometheus-compatible replacement with better compression, faster ingestion, and built-in clustering. Simpler operationally than Thanos.

Cortex / Mimir - horizontally scalable, multi-tenant Prometheus. Standard in large organizations using Grafana Cloud.

Cardinality management¶

High cardinality destroys Prometheus performance. The most common causes:

labels with unbounded values: user IDs, request IDs, IP addresses, pod names with hashes
recording every HTTP path as a label (use pattern matching or drop high-cardinality paths)
short-lived jobs pushing to Pushgateway without cleanup

# Find high-cardinality metrics
curl -sg 'http://prometheus:9090/api/v1/label/__name__/values' | jq '.data | length'
curl -sg 'http://prometheus:9090/api/v1/query?query=topk(10,count by (__name__)({__name__=~".+"}))' \
  | jq '.data.result[] | {metric: .metric.__name__, count: .value[1]}'

Drop unnecessary labels at scrape time with relabelings in the ServiceMonitor.

Native Kubernetes service discovery¶

The Operator's ServiceMonitor/PodMonitor CRDs are a convenience layer - underneath, they compile down to kubernetes_sd_configs in the Prometheus config file. If you're not running the Operator (or you need an SD role the CRDs don't expose), you write these directly.

scrape_configs:
  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: "true"
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod

The main role values you'll use:

Role	Discovers	Common use
`node`	one target per cluster node	node-exporter, kubelet self-metrics
`pod`	one target per pod, per declared container port	annotation-based scraping, sidecars
`service`	one target per Service port	blackbox probing, service-level scraping
`endpoints`	one target per backing pod behind a Service	the role ServiceMonitor actually compiles to
`ingress`	one target per Ingress rule	external blackbox checks against Ingress hosts

`relabel_configs` vs `metric_relabel_configs`¶

This is one of the most commonly confused parts of the Prometheus config, because both blocks look identical and both can add, drop, or rename labels. The difference is when they run:

relabel_configs runs before the scrape, against the SD-provided meta labels (__meta_kubernetes_*, __address__, __scheme__). It decides which targets get scraped at all and how to reach them. Dropping a target here means Prometheus never makes the HTTP request.
metric_relabel_configs runs after the scrape, against the labels of the metrics that came back in the response body. It decides which of the already-fetched samples get ingested into the TSDB. The scrape still happened - you're discarding data after paying the network and parse cost.

scrape_configs:
  - job_name: node-metrics
    kubernetes_sd_configs:
      - role: node
    relabel_configs:
      # only scrape nodes in the "worker" pool -- targets never even get hit
      - source_labels: [__meta_kubernetes_node_label_node_role]
        regex: worker
        action: keep
    metric_relabel_configs:
      # the target IS scraped, but we throw away noisy per-cpu metrics
      # after the fact because they're not worth the cardinality
      - source_labels: [__name__]
        regex: "node_cpu_seconds_total"
        action: drop

Use relabel_configs to control cost at the network/target level (fewer scrapes, smaller target list). Use metric_relabel_configs to control cost at the cardinality level (fewer series stored per scrape) - for example dropping a vendor exporter's high-cardinality metrics you don't control the instrumentation of.

Federation vs remote-write¶

Prometheus offers two fundamentally different ways to build a multi-Prometheus topology, and they solve different problems.

Federation (/federate endpoint): a "global" Prometheus scrapes a curated subset of time series from other Prometheus servers, using the same pull model as any other target.

scrape_configs:
  - job_name: federate
    honor_labels: true
    metrics_path: /federate
    params:
      "match[]":
        - '{__name__=~"job:.*"}'   # federate only recording-rule rollups
    static_configs:
      - targets:
          - prometheus-cluster-a:9090
          - prometheus-cluster-b:9090

Federation is hierarchical and pull-based: the global Prometheus has to reach every leaf Prometheus over the network, and it only sees what you explicitly match[] on. It works fine for a small number of clusters and a small number of aggregate series, but it does not scale to raw metric retention - you're meant to federate rollups (recording rule output), not everything.

Remote-write: each leaf Prometheus pushes its samples to a central system (Thanos Receive, Cortex, Mimir, VictoriaMetrics) as they're scraped, over the WAL-based remote-write protocol.

remote_write:
  - url: https://mimir.example.com/api/v1/push
    queue_config:
      capacity: 10000          # samples buffered per shard before blocking
      max_shards: 30           # parallel send workers
      min_shards: 1
      max_samples_per_send: 2000
      batch_send_deadline: 5s
    metadata_config:
      send: true

This is why remote-write has largely won for multi-cluster and long-term storage: it's push-based (no need to open inbound network paths to every leaf cluster), every raw sample is available centrally (not just what you thought to federate), and it decouples local retention (fast, cheap, short-lived TSDB per cluster) from long-term retention (object storage, downsampling, global query). Federation still has a place for small, low-cardinality "mission control" dashboards that aggregate a handful of clusters, but treat it as the exception, not the default.

Remote-Write 2.0 is a rewritten wire protocol (based on protobuf with string interning and native support for metadata, exemplars, and native histograms in a single request) that fixes a lot of Remote-Write 1.0's inefficiency. As of Prometheus 3.x it's implemented and can be negotiated between a 2.0-aware sender and receiver, but the spec is still marked experimental upstream and 1.0 remains the default/fallback protocol - check that both your Prometheus version and your remote-write receiver (Mimir, Thanos Receive, VictoriaMetrics, etc.) support 2.0 before relying on it.

HA Prometheus pairs¶

Because Prometheus is a single-process, locally-persistent system, high availability means running two (or more) identical Prometheus replicas that scrape the same targets with the same config, rather than clustering a single logical Prometheus.

flowchart LR
    Targets[Scrape targets] --> P1[Prometheus replica A]
    Targets --> P2[Prometheus replica B]
    P1 --> AM1[Alertmanager 1]
    P2 --> AM2[Alertmanager 2]
    AM1 <-->|gossip protocol| AM2
    AM1 --> Notify[Single notification\nper firing alert]
    AM2 --> Notify

Both replicas evaluate the same alerting rules independently and both fire the same alert to Alertmanager. Alertmanager instances form a gossip cluster (using the same protocol family as Serf/memberlist) and deduplicate: even though two Alertmanagers each receive the same alert from their respective Prometheus, the cluster agrees on a single notification. Point both Prometheus replicas at the same list of Alertmanager addresses so this dedup actually happens:

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager-0.alertmanager-headless:9093
            - alertmanager-1.alertmanager-headless:9093
            - alertmanager-2.alertmanager-headless:9093

The tradeoff of HA pairs: you get resilience against a single Prometheus pod dying, but the two replicas' local TSDBs are independent - they can drift slightly in what they've scraped (a missed scrape on one but not the other), and dashboards querying a single replica directly can show gaps that don't exist on the other. Query through a layer that can merge/dedupe both replicas (Thanos Query with --query.replica-label=replica, or a load balancer with retry) rather than pointing Grafana at one replica.

Remote-write tuning and backpressure¶

Remote-write is WAL-based: Prometheus writes every scraped sample to its write-ahead log first, then a separate reader tails the WAL and ships samples to configured remote endpoints in parallel shards. If the remote endpoint can't keep up, samples queue in memory per shard, and Prometheus scales max_shards up to compensate - up to the configured ceiling.

Symptoms of remote-write backpressure:

prometheus_remote_storage_samples_pending climbing steadily instead of hovering near zero
prometheus_remote_storage_shards pinned at max_shards continuously (it can't add more parallelism)
rising prometheus_remote_storage_samples_dropped_total or ..._failed_total - once queues are full, Prometheus drops samples rather than blocking scrapes
prometheus_remote_storage_highest_timestamp_in_seconds minus prometheus_remote_storage_queue_highest_sent_timestamp_seconds widening - the gap is now-vs-sent lag

The fix is rarely "just raise max_shards further" - it's usually that the receiving system (Mimir ingesters, Thanos Receive) is the actual bottleneck and needs more replicas, or that max_samples_per_send/batch_send_deadline are tuned for a network path with too much latency per round trip. Also check min_backoff/max_backoff under queue_config - overly aggressive backoff on transient errors compounds queue growth during a receiver blip.

Exemplars: bridging metrics and traces¶

Exemplars attach a trace ID (and optionally other high-cardinality context) to an individual metric sample, without that context becoming a label on the time series itself. A histogram bucket stores its normal float value plus an attached exemplar like trace_id="abc123", so from a latency spike in a graph you can jump directly to the trace that produced one of the samples in that bucket.

# prometheus.yml -- exemplar storage must be enabled explicitly
storage:
  exemplars:
    max_exemplars: 100000

# --enable-feature=exemplar-storage on the Prometheus binary/Operator spec

Client libraries (via OpenTelemetry or the Prometheus client libraries with exemplar support) attach the exemplar at instrumentation time, typically pulling the active trace ID from context. Grafana renders exemplars as small diamond markers on top of histogram panels - clicking one deep-links into the trace in Jaeger or Tempo. This is the practical answer to "how do metrics and tracing actually connect": metrics tell you that something is slow, exemplars tell you which specific request to go look at in a trace to find out why.

Advanced PromQL¶

Subqueries¶

A subquery evaluates a range vector expression as if it were itself sampled over time - useful for smoothing an already-rated metric or computing max/min over a rate.

# max value of the 5m request rate, evaluated every 1m, over the last hour
max_over_time(rate(http_requests_total[5m])[1h:1m])

Subqueries are expensive - the inner expression is re-evaluated at every step of the outer range - so use them deliberately, not as a default habit.

Detecting absence¶

absent() and absent_over_time() are the most under-used alerting primitives in PromQL. A missing time series is invisible to every other function - rate() on a metric that stopped existing simply returns nothing, so a naive alert silently stops firing instead of firing "job is down."

# fires if no time series matching this selector exists right now
absent(up{job="payment-api"})

# fires if the metric existed at some point but has produced
# no samples for the last 10 minutes -- catches "stopped scraping"
# distinct from "never existed"
absent_over_time(up{job="payment-api"}[10m])

Wrap every "critical dependency must be exporting metrics" alert with an absent_over_time companion. Scrape target failures, exporter crashes, and misconfigured relabeling that accidentally drops a job are all cases where the absence of a metric is the actual incident, and they're exactly the cases that a threshold-based alert on the (now nonexistent) metric will never catch.

`group_left` / `group_right` and the cardinality trap¶

sum by (...) aggregations collapse cardinality, but many-to-one joins do the opposite - they multiply it if you're not careful. group_left/group_right let you join a vector with a many-to-one (or one-to-many) relationship, most commonly to enrich a metric with business labels that live only on kube_pod_labels or a similar info metric.

# enrich per-pod CPU usage with the "team" label that only exists
# on kube_pod_labels (one series per pod, many labels)
sum by (pod, namespace, team) (
  rate(container_cpu_usage_seconds_total{container!=""}[5m])
  * on (pod, namespace) group_left(team)
    kube_pod_labels{label_team!=""}
)

group_left(team) says "the left side (CPU usage) is many series per matched (pod, namespace), the right side (kube_pod_labels) is one - pull the extra team label from the one side onto the many side." Get the direction backwards (group_right when you meant group_left) and the query either errors with "multiple matches for labels" or, worse, silently produces a cross-product explosion if the match labels aren't as unique as you assumed.

The real danger: if kube_pod_labels (or whatever you're joining against) has more than one series per match key - for instance because a pod was recreated and both the old and new series are still within the query window, or because a label used in on(...) isn't actually unique per pod - the join produces one output series per combination. This is a common way a well-behaved dashboard query suddenly explodes cardinality and slows a whole Prometheus instance to a crawl. Always match on the narrowest possible key (pod, namespace, and ideally something like uid), and prefer on(...) over the default (all shared labels) so the join surface is explicit and reviewable.

Multi-window, multi-burn-rate SLO alerting¶

Static thresholds ("alert if error rate > 5%") are a blunt instrument: they don't account for how fast you're burning your error budget, so they either fire too late on a slow leak or fire immediately (and noisily) on a brief blip that self-resolves. The pattern from the Google SRE workbook - multi-window, multi-burn-rate alerting - fires based on how quickly the SLO's error budget is being consumed, checked across both a short and a long window so a real fast-burn incident pages immediately while a short blip doesn't.

The core idea: define a burn rate as (actual error rate) / (error budget for the SLO period). A burn rate of 1 means you're consuming the budget exactly as fast as the SLO allows over its full window; a burn rate of 14.4 means you'd exhaust a 30-day budget in about 2 days.

groups:
  - name: slo.burn-rate
    rules:
      - alert: HighBurnRateFastWindow
        expr: |
          (
            sum(rate(http_requests_total{job="api",status=~"5.."}[1h]))
            / sum(rate(http_requests_total{job="api"}[1h]))
          ) > (14.4 * 0.001)
          and
          (
            sum(rate(http_requests_total{job="api",status=~"5.."}[5m]))
            / sum(rate(http_requests_total{job="api"}[5m]))
          ) > (14.4 * 0.001)
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Fast burn: exhausting error budget in ~2 days at current rate"

      - alert: HighBurnRateSlowWindow
        expr: |
          (
            sum(rate(http_requests_total{job="api",status=~"5.."}[6h]))
            / sum(rate(http_requests_total{job="api"}[6h]))
          ) > (6 * 0.001)
          and
          (
            sum(rate(http_requests_total{job="api",status=~"5.."}[30m]))
            / sum(rate(http_requests_total{job="api"}[30m]))
          ) > (6 * 0.001)
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Slow burn: exhausting error budget in ~5 days at current rate"

Here 0.001 is the target error rate for a 99.9% SLO. Requiring both a long window and a short window to exceed the burn-rate threshold before firing is what makes this pattern resilient to noise: a 5-minute blip that recovers won't sustain the 1h/6h long-window condition, but a genuine fast-burn incident satisfies both windows almost immediately. Pair a fast/short pairing (page immediately, short time-to-detect) with a slow/long pairing (ticket-only, catches slow leaks that never spike hard enough to trip the fast alert). This is meaningfully better than a flat > 0.05 threshold because it directly answers the question an SLO alert should answer: "at this rate, when do we breach the SLO, and does a human need to act now or can it wait."

Common production mistakes¶

Alerting on raw counters instead of rate(). A counter alert like http_errors_total > 100 fires once and then never resets meaningfully - counters only go up. Always wrap counters in rate() or increase() before comparing against a threshold.
No absent() companion alerts. If the exporter crashes, the scrape target disappears, or relabeling accidentally drops a job, threshold alerts on that metric go silent instead of firing. Treat "no data" as its own alertable condition for every critical target.
Confusing relabel_configs with metric_relabel_configs. Using the pre-scrape block to drop unwanted metric names does nothing (those meta labels don't exist yet), and using the post-scrape block to filter targets wastes a full scrape for data you throw away.
Unbounded label cardinality via joins. A group_left join against an info metric with more than one series per match key silently multiplies cardinality. Always constrain on(...) to a narrow, genuinely unique key.
Dashboards querying a single HA replica directly. Point Grafana (or any consumer) at a query layer that merges/dedupes HA pairs, not at one replica's :9090 directly - otherwise dashboards show gaps that don't reflect real data loss.
Federating raw metrics instead of rollups. /federate without a tight match[] against recording-rule output turns into a second full-cardinality scrape of every leaf Prometheus, defeating the point of federation and often falling over at scale.
Ignoring remote-write queue metrics. prometheus_remote_storage_samples_pending/..._dropped_total are the first sign of a receiver-side bottleneck; by the time users notice missing data in Grafana, samples have already been silently dropped for a while.
Static error-rate thresholds for SLO alerting. A flat > 5% threshold either misses slow leaks or pages on noise. Multi-window, multi-burn-rate alerting answers the actually useful question - "will we breach the SLO, and how urgently."

Source Links¶

OpenTelemetry - vendor-neutral instrumentation and the exemplar/trace bridge into Prometheus
Jaeger - where trace data referenced by exemplars actually lives
Falco
Istio
Operations and Maintenance
Audit and Logging