API Quotas, Fairness & Backpressure

Quotas, Fairness, and Backpressure

A rate limiter that allows 1,000 requests per second for everyone together protects a service from collapse — and lets one customer's runaway script consume all 1,000, so every other customer sees errors. Shared systems need more than a single limit. They need quotas that reflect what each tenant paid for, fairness so no tenant can starve others, and backpressure so that when the system is genuinely overloaded it slows or rejects work early and clearly, instead of queuing until every request times out.

This page covers the different kinds of limits, how to apply them per tenant, how to shed load safely, and how to tell clients what to do when they are limited.

TL;DR

Rate limits cap requests per time window; quotas cap usage per billing period; concurrency limits cap work in flight.
Apply limits per tenant (and per user, key, or endpoint) with token buckets, plus a global limit protecting the service.
Use fair queuing, priority tiers, and shuffle sharding so a noisy tenant affects as few others as possible.
Under overload, shed load early — reject or drop work before it consumes resources, and drop requests that have already exceeded their deadlines.
Respond with 429 (client over its limit) or 503 (service overloaded), always with Retry-After.
Clients must back off with jitter, respect Retry-After, and use retry budgets.

Quick Example

A per-tenant token bucket implemented atomically in Redis with a Lua script:

When a tenant runs out, the API tells the client exactly when to come back:

Expensive endpoints can charge a higher cost per call, so one bucket fairly meters cheap and expensive requests.

Core Concepts

Kinds of Limits

Combine them: per-tenant rate limits and quotas for fairness and billing, per-tenant concurrency limits for expensive operations, and a global limit for the service's real capacity. See Rate Limiting.

Fairness Between Tenants

Per-tenant buckets give each tenant its own allowance, sized by plan.
Weighted fair queuing serves queued work from tenants in turn, weighted by tier, so a tenant with 10,000 queued jobs does not delay a tenant with 10.
Priority tiers let interactive traffic bypass bulk work under contention.
Shuffle sharding assigns each tenant a small random subset of workers or partitions, so a tenant that overwhelms its shard affects few others.
Separate pools for very large tenants prevent them from dominating shared capacity.

See Tenant Isolation and Noisy Neighbor Testing.

Backpressure and Load Shedding

When demand exceeds capacity, something must give. The worst outcome is unbounded queuing: latency grows until every request times out, so the system does full work for responses nobody receives. Better strategies:

Bounded queues with rejection when full.
Deadline propagation: pass the caller's deadline downstream, and drop work whose deadline has passed instead of processing it.
Queue timeouts and LIFO under overload so fresh requests succeed while stale ones are dropped.
Adaptive concurrency limits that measure latency and lower the allowed concurrency as it rises, in the spirit of TCP congestion control.
Shedding by priority: drop background and low-priority work first.

For asynchronous systems, backpressure means consumers pull at their own pace and producers are slowed or rejected when queues exceed their limits.

Telling Clients What Happened

Include a machine-readable problem body, and document limits per plan so customers can design within them.

Client Behavior

Server-side limits only work if clients cooperate:

Respect Retry-After exactly.
Exponential backoff with jitter for other retries, so clients do not retry in synchronized waves.
Retry budgets: cap retries as a fraction of requests, so retries cannot multiply load during an outage.
Retry only idempotent requests, or use idempotency keys. See Idempotency.
Circuit breakers stop calling a dependency that is failing. See Circuit Breakers.

Best Practices

Limit at the Edge and in the Service

Gateways can enforce coarse per-key limits cheaply; services enforce limits that depend on business context, such as cost per operation or tenant plan. See API Gateway.

Size Limits From Capacity Tests

Measure what the service handles before latency degrades, and set global limits below that point, with headroom for failover.

Observe Limiting

Record every limited request with tenant, limit name, and endpoint. Dashboards of who is being limited — and how close others are — guide plan changes and abuse investigations.

Fail Open or Closed Deliberately

Decide what happens if the rate limiter's store (for example Redis) is unavailable: allowing traffic risks overload; blocking it causes an outage. Local fallback limits are a common compromise.

Common Mistakes

One Global Limit

Unbounded Queues Under Load

Retry Storms

FAQ

What is the difference between a rate limit and a quota?

A rate limit caps how fast requests arrive — for example 100 per minute. A quota caps total usage over a longer period — for example 1 million calls per month — usually tied to a plan or budget.

How do I make rate limiting fair across tenants?

Give each tenant its own limits sized by plan, use fair queuing for shared work queues, add priority tiers for interactive traffic, and isolate the largest tenants with dedicated capacity or shuffle sharding.

Should I return 429 or 503?

Return 429 when a specific client exceeded its limit. Return 503 when the service as a whole is overloaded or unavailable. Include Retry-After in both.

What is load shedding?

Deliberately rejecting or dropping some work when demand exceeds capacity — ideally the lowest-priority or already-expired work — so the remaining requests complete quickly instead of all timing out.

What is backpressure?

A signal from a slower component to faster ones to slow down: bounded queues that reject when full, consumers that pull at their own pace, or responses that tell clients to retry later.