Metric Cardinality & Telemetry Cost Control

Metric Cardinality and Telemetry Costs

Observability bills have a way of growing faster than the systems they observe. A developer adds user_id as a metric label, and one counter becomes two million time series. A deployment changes pod names every hour, and series churn exhausts Prometheus memory. Debug logging left on in production doubles log ingestion overnight. Traces are kept at 100% for a high-traffic endpoint nobody looks at. Each change is reasonable locally; together they make telemetry one of the largest infrastructure costs — or take down the monitoring system during the incident it was meant to help with.

The central concept is cardinality: the number of unique time series a metric produces. Managing it, along with log volume and trace sampling, keeps observability affordable and reliable.

TL;DR

Cardinality is the number of unique label combinations; it multiplies across labels.
Never use unbounded values — user IDs, request IDs, raw URLs, error messages — as metric labels.
Churn (labels that change, like pod names in short-lived jobs) creates series even when concurrent counts look small.
Find offenders with TSDB status pages and count by (__name__) queries; set label budgets per metric.
Control at the source and in pipelines: route templates, relabeling, dropping, recording rules, and histogram bucket choices.
Reduce logs and traces with levels, sampling, and filtering in the Collector, and alert on telemetry growth.

Quick Example

Find which metrics and labels produce the most series (PromQL):

Protect Prometheus at scrape time and strip dangerous labels:

And fix the cause in instrumentation — label by route template, not raw path:

Core Concepts

How Cardinality Multiplies

A metric's series count is roughly the product of each label's distinct values:

Every label added to a widely used metric multiplies its cost. Memory, storage, query time, and vendor pricing (often per active series or per custom metric) all scale with series counts.

High-Cardinality and Churning Labels

Identifiers belong in logs and traces, which are designed for per-event detail; exemplars can link a metric sample to a trace ID without adding a label.

Histograms

Classic histograms create one series per bucket per label combination. Choose bucket boundaries around your SLO thresholds rather than accepting large defaults, avoid splitting histograms by high-cardinality labels, and consider native (exponential) histograms, which represent distributions far more compactly. See Prometheus Metric Types.

Recording Rules and Aggregation

Precompute what dashboards need — for example request rates per service, without instance labels — with recording rules or streaming aggregation, then drop or shorten retention for the raw high-cardinality series.

Logs and Traces

See OpenTelemetry Collector and OpenTelemetry Sampling.

Budgets and Governance

Label budgets: document allowed labels and maximum distinct values for shared metrics; review new labels in code review.
Per-team quotas: allocate series and ingestion budgets per team or service, with usage dashboards.
Scrape limits: sample_limit and label_limit per job stop a single bad deploy from overwhelming the system.
Alerts on growth: alert when active series, ingestion rate, or log volume jumps compared with the previous week.
Cost attribution: show telemetry costs per team alongside other cloud costs. See Cloud Costs.

Best Practices

Decide Labels at Design Time

For each metric, ask which questions it must answer and add only the labels needed to slice those answers.

Keep Identifiers Out of Metrics

Put per-request and per-user details in logs and traces, linked through trace IDs and exemplars.

Normalize Before You Label

Use route templates, status classes (2xx, 5xx), and bounded enumerations instead of raw values.

Review Telemetry Regularly

Periodically remove unused metrics and dashboards, shorten retention for low-value data, and re-check sampling rates.

Common Mistakes

Customer ID as a Metric Label

Pod Names on Batch Job Metrics

Debug Logging Left On

FAQ

What is metric cardinality?

The number of unique time series a metric produces, determined by the combinations of its label values. High cardinality increases memory, storage, query cost, and often vendor charges.

Why is high cardinality a problem?

Each series consumes memory and storage in the monitoring system. Labels with many or unbounded values multiply series counts, slowing queries, raising costs, and potentially crashing the monitoring system.

How do I find high-cardinality metrics in Prometheus?

Use the TSDB status page or API, which lists metrics and labels with the most series, and PromQL queries such as topk(10, count by (__name__) ({__name__=~".+"})).

Where should user IDs and request IDs go?

In logs and traces, which are designed for per-event detail. Metrics should use bounded labels, and exemplars can connect metric samples to specific traces.

How do I reduce log and trace costs?

Use appropriate log levels, drop noisy logs, sample repetitive events, apply head and tail sampling to traces, process telemetry in an OpenTelemetry Collector before export, and use tiered retention.