Grafana Loki: Labels, LogQL, Storage, and Retention

Grafana Loki

Grafana Loki organizes logs into labeled streams and lets you search them with LogQL. Begin with a few stable labels that identify where logs came from, then keep request-specific detail in the log record or suitable structured metadata. Turning every request ID into a stream label gives your index a very busy social life.

The practical decision is whether your collection, query, storage, and retention design can answer the investigations you need at an acceptable operating cost. Choosing a logging backend alone does not establish that evidence will arrive, remain available, or be safe to expose.

TL;DR

Choose low-cardinality source labels before shipping a large log volume.
Narrow streams and time ranges before parsing or searching individual lines.
Use a supported collector and test timestamp, parsing, redaction, and retry behavior.
Plan object storage, index storage, and retention together.
Verify delivery and query visibility using known records, including a failed delivery case.
Measure representative investigations instead of assuming a smaller index guarantees cheaper queries.

Quick Example

This complete Python fixture creates a Loki JSON push payload locally. It prints fictional records and sends nothing. Save it as example.py and run with Python 3.12; the generated timestamps use the current time so a later ingestion pilot can query an appropriate window.

The Loki HTTP API defines the push format and requires JSON push timestamps to be strings. This fixture checks construction, not server acceptance. To extend it, send only these fictional records through an authorized test collector or test endpoint with its required authentication and tenant configuration.

After successful ingestion, try this LogQL query over the matching time range:

Expect the req-2 record if the collector preserved its JSON fields and labels. The query has not been executed against Loki here. The log-query reference explains stream selectors and sequential parsing/filter stages. A locally generated payload is not evidence of query visibility.

Core Concepts

Labels identify streams

Loki indexes the label sets associated with streams rather than the full text of every line. Labels should describe relatively stable source dimensions; frequent unique values belong elsewhere. See Loki label guidance. A request ID is valuable investigation context without needing to be an indexed source dimension.

Cardinality combines across dimensions

The relevant quantity is the distinct combination of label values. For a fictional service with three environments, four regions, and five instances per environment-region pair, the maximum combination count is 3 × 4 × 5 = 60 if all combinations exist. Adding a unique label to every request can produce a new stream per request. This arithmetic is an illustration, not a measured limit or capacity recommendation.

Query stages do different work

The selector identifies streams; subsequent stages inspect or transform their lines. A narrow selector and time window reduce the candidate data before expensive parsing. Put the query beside its intended question: “find this request's error” and “count all errors over a month” are different workloads even when they begin with the same service label.

Follow a Record Through the Pipeline

Use this as a proposed pilot route. Repeat a known log emission while temporarily interrupting the test destination, then inspect retry behavior and the final record count. Do not infer lossless delivery from one successful response, or duplicate suppression from one visible record. Also check shutdown and collector restart behavior.

For a new collection setup, use Grafana Alloy or another supported client. The official Promtail notice states that Promtail reached end of life on March 2, 2026. Existing configuration examples that begin with Promtail need a migration decision, not automatic reuse.

Storage and Retention Are One Design

Loki stores compressed log chunks and an index that locates them. The storage guide describes TSDB index storage and supported object stores; a local filesystem experiment does not demonstrate production durability or scaling. Review the actual deployment mode, object-store permissions, capacity, and recovery route together.

The retention guide describes Compactor-based retention and warns that bucket lifecycle policies must outlast the configured retention period. Verify that retention is enabled and actually takes effect. An object-store expiration rule that removes data prematurely can defeat the query window you intended to keep.

Choose a retention period from investigation requirements and data sensitivity, then estimate ingest volume, storage, requests, and query compute using representative logs. Keeping everything forever is an operating policy with consequences, not the absence of a policy. Keep sensitive fields out of emitted records where possible and verify transformations before sending real logs.

Best Practices

Start from investigation questions

Collect several representative queries before selecting labels. Stable application and environment labels may help a service investigation; a user identifier can stay in a record or appropriate metadata. Revisit the choice when actual query paths show a missing useful source dimension.

Test malformed and delayed records

Mix valid JSON, malformed lines, delayed timestamps, and a destination outage in an isolated pilot. Record accepted, rejected, retried, and visible counts. This reveals where evidence disappears instead of attributing every search miss to the query language.

Keep logging and access boundaries explicit

Test collector credentials, tenant routing, read access, and redaction with fictional records. A correlation ID can still reveal a sensitive workflow when combined with other data. Give operators the visibility needed for support without treating the log store as public documentation.

Common Mistakes

Bad: Promote every parsed field to an indexed stream label.

Correct: Keep a deliberate low-cardinality label set and inspect other fields during queries or through supported metadata features.

Bad: Assume a log exists because the application printed it.

Correct: Follow a known record through collection, endpoint acceptance, and query visibility.

Bad: Apply a bucket expiration rule without coordinating Loki retention.

Correct: Review both lifecycle mechanisms and verify the intended investigation window in a controlled test.

FAQ

Is Loki a full-text index for logs?

Its core stream index uses labels. Query stages inspect the selected log content. Match your investigation workload to that storage and query model rather than assuming another backend's index behavior.

Should request IDs be labels?

Usually keep high-cardinality request identifiers in the record or suitable structured metadata. Test the actual query requirement before adding them to indexed source dimensions.

Does a smaller index guarantee lower total cost?

No. Ingest processing, object storage, query work, retention, availability, and operator effort also matter. This article makes no measured price or performance comparison.

What can I check before deploying a Loki pipeline?

That records stay structured JSON, timestamps parse, and high-cardinality values such as request IDs stay out of labels. Collector delivery, LogQL queries, retention, access control, and recovery need a real integration pilot.