LLM Latency, Cost, and Capacity: Measure Useful Work

LLM Latency, Cost, and Capacity Planning

LLM latency, cost, and capacity planning starts with a useful completed task under a stated latency objective. Count input and output tokens, waiting time, failed attempts, and quality failures together. A cheap response that arrives too late or cannot answer the question still consumed capacity.

Choose a representative request mix before choosing a server count. Short cached prompts, long uncached prompts, and long generations stress different parts of the serving path. A single average request hides the queue you will eventually have to explain.

TL;DR

Separate time to first token, generation pace, end-to-end completion, and client-side waiting.
Measure offered load and successful work under the same latency and quality rules.
Track prompt/output length distributions and cache-hit evidence.
Count retries, failed attempts, and non-token operating costs in the estimate.
Run a concurrency sweep with representative inputs before declaring capacity.

Quick Example

This complete Python 3.12 calculation uses fictional rates and four invented requests. It calls no model or provider. Rates are currency units per million tokens, chosen only to illustrate the denominator.

Expect 0.0077 per useful task. This excludes infrastructure, retrieval, tools, storage, cached-token rate differences, and other charges. It is a declared arithmetic model, not provider billing or a measured model benchmark. A real evaluation must also distinguish transport errors from quality failures.

Core Concepts

Latency has several clocks

Time to first token measures the wait until generation begins; it does not establish when a usable answer is complete. Track arrival, admission, first output, last output, and any tool round trips. Report percentiles on the relevant request population, including failures and timeouts in a separately visible count. The vLLM serving benchmark reference distinguishes first-token, inter-token, end-to-end, queue, and goodput measurements.

Capacity is conditional

State the model, hardware or service tier, request distribution, concurrency, and quality objective with a result. The same token throughput may correspond to very different user waits. A deployment serving mostly short requests does not inherit the long-context capacity of a different benchmark configuration.

Caches change the workload

Keep cold and warm cases separate, and record actual hit evidence rather than assuming repeated text is cached. Prefix reuse, application response caching, and retrieval caches protect different boundaries. The vLLM prefix-caching guide explains reuse of computation for shared prefixes; it does not make arbitrary generated answers free or interchangeable.

A Capacity Experiment That Can Fail

Begin below expected peak and increase offered load in controlled steps. Keep arrival rate separate from in-flight concurrency. An overloaded client that waits to send new requests can make a server look stable by reducing the offered load. Record the load-generation method and where waiting is included.

For a self-hosted pilot, observe accelerator memory, scheduling, and utilization alongside request outcomes. For a hosted service, observe account limits, throttling, retry headers, and request usage. Neither arrangement removes the need to bound queues and protect upstream callers from indefinite waiting.

Compare configurations by useful tasks per time window and total cost under the same rules. A faster but less accurate configuration can lose after retries or human repair. If quality cannot yet be judged, label the result as serving performance and keep task-success conclusions pending.

Best Practices

Preserve a workload manifest

Record input/output length distributions, model/version, generation settings, cache state, concurrency, and evaluation criteria. Use synthetic or approved data with representative structure. A benchmark you cannot reproduce is weak evidence for a capacity purchase.

Budget for unsuccessful work

Count charged attempts, timeouts, retries, and tool calls. Track both cost per attempted task and cost per useful task. If success approaches zero, report the failure instead of dividing by a convenient population that excludes it.

Define overload behavior

Choose admission, queue bounds, timeouts, cancellation, and a controlled degradation policy. Rehearse a burst and recovery after it ends. Scaling may take time, so autoscaling alone is not the overload policy.

Common Mistakes

Bad: Compare token throughput without stating request lengths.

Correct: Hold the workload and acceptance rules constant and report their distribution.

Bad: Treat a cached prompt assumption as a measured saving.

Correct: Verify the cache mechanism and hit conditions, then measure cold and warm paths.

Bad: Divide cost only by responses that returned HTTP success.

Correct: Apply the declared latency and task-quality rules and retain failed attempts in the numerator.

FAQ

Does streaming eliminate latency?

It can expose early output, while complete-answer and tool latency remain. Test the interaction the user actually needs.

Can token counts alone determine server count?

They are inputs to the experiment. Queueing, memory, model behavior, hardware, and request mix also determine usable capacity.

Are the example rates current prices?

No. They are explicitly fictional inputs for arithmetic, with no provider or purchasing claim.

What should change the plan?

A representative load sweep that misses latency, quality, recovery, or cost objectives should change the configuration, workload policy, or capacity estimate.