LLM Provider Outages and Fallback Behavior
Define fallback by the task the application must complete, including output meaning, allowed data handling, tool authority and failure behavior. A second provider is useful only if it can satisfy that contract. A successful response from somewhere else does not automatically preserve quality, privacy or completed-effect semantics.
Separate failures before visible output, failures during a stream, and failures after a tool or external effect may have occurred. Those states need different recovery decisions. Establish a total attempt/time budget and an honest terminal outcome before adding more routes.
TL;DR
- Define fallback by the task contract — output meaning, allowed data handling, tool authority, and failure behavior — not by whether another provider answers.
- A second provider helps only if it satisfies that contract; a successful response elsewhere can still change quality, privacy, or side effects.
- Treat failures before output, mid-stream, and after a tool or external effect as different states with different recovery rules.
- Set a total attempt and time budget and an honest terminal outcome before adding more routes.
- Classify errors before retrying, and test fallbacks against the task's required properties, not just "returned text."
Quick Example
This Python function chooses among three policy labels. It makes no provider request and parses no streaming protocol. The caller must already have classified the failure and established that the fallback is approved and compatible.
The model refuses an automatic switch after output was emitted, rejects a bad-request category, and stops when the attempt budget is exhausted. These are design choices for this example. It does not implement time budgets, rate-limit classification, retry delays, tool reconciliation or a quality evaluator. It cannot establish actual provider availability or compatibility.
Core Concepts
An error category is evidence, not a universal remedy
Read the chosen provider's current error contract. As one concrete example, Claude's error documentation describes streaming errors after an HTTP 200 response and distinguishes rate-limit-related conditions. Do not generalize one provider's status handling to every API or assume every 429 is a short transient event. Claude API errors
Classify invalid input, authorization, quota, temporary overload, transport timeout and malformed output separately. A retry may repeat a permanent error or conceal a configuration issue. The fixture's rate_limit input is an abstract category after that review, not an instruction to retry every real 429.
Retry and fallback consume a shared budget
AWS's retry guidance explains how repeated attempts can amplify overload and why bounded retries, backoff and jitter need deliberate placement. Apply that reasoning to the whole request path rather than allowing every library and service layer to retry independently. Timeouts and retries
Track the overall deadline, remaining attempts and work already performed. A fallback that starts after the user has stopped waiting can still consume resources or trigger effects. Decide how cancellation propagates and what outcome remains observable when the total budget is exhausted.
This is an editorial test matrix. None of the actual network, model, tool or user-interface branches were exercised by the policy function.
Partial streams change the user contract
Once text is visible, silently replacing the generator can produce a response with mismatched reasoning, repeated content or contradictory instructions. Decide whether the interface stops and reports interruption, offers a clear restart, or uses a continuation method that has been explicitly validated for the task.
Structured responses need their own completion and validation boundary. A partially streamed object or tool argument should not automatically become an executable instruction. Wait for the protocol's required completion evidence and validate against the application contract before using the result.
A tool call is not merely more text
If a request can invoke an external action, define its durable operation identity and outcome lookup. A provider timeout does not reveal whether a downstream action completed. Repeating the whole task through another provider can repeat that effect unless the application controls identity and reconciliation independently.
Keep the model's output outside the authority decision. An alternative provider must not receive broader tool permissions merely because the primary route failed. Preserve user intent, approved scope and server-side enforcement across every route.
Quality belongs in fallback acceptance
Use a shared task suite with representative inputs, expected constraints and failure criteria. Evaluate the actual fallback's output validity, correctness, language behavior, tool compatibility and operating limits. A fast response that violates a required format is a failed fallback for that contract.
Review data handling separately. Confirm which content may be sent to each approved destination and which contractual or regional restrictions apply. A provider outage does not create authorization to transmit restricted data somewhere new. No private prompts or provider credentials were used for this article.
Best Practices
Build deterministic failure fixtures around the client boundary: timeout before output, rate-limit classification, disconnect after partial output, malformed final structure and ambiguous tool outcome. Verify the application's visible result and absence of unintended effects. A mocked failure tests application logic, while a controlled integration test supplies additional evidence about the real SDK and protocol.
Record route, failure class, attempts, elapsed budget and terminal outcome using approved telemetry. Avoid logging complete private prompts, outputs or credentials merely to count fallback attempts. Include a quality signal where the task permits it so availability does not hide degradation.
Test route recovery as well as route failure. Define when the application returns to the primary and how it avoids oscillating between unhealthy paths. A circuit or routing policy introduces state that needs tests of its own; it is not automatically safer because it has a familiar name.
Reverse an automatic-fallback plan when the alternative lacks a required capability or approved data boundary. An explicit unavailable result may be the correct contract-preserving outcome. Add a provider only after the shared task suite and operating review show what it can safely replace.
Common Mistakes
Treating HTTP 200 as a completed stream
Retrying every failure the same way
Stitching partial output from different models
Testing only that a fallback returned text
FAQ
Was a provider outage tested live?
No. The local policy assertions test the explicit toy decisions only. Actual SDK, transport, stream, model quality and routing behavior remain untested.
Can we switch providers after a partial response?
Only through a deliberate, tested user and task contract. The example chooses to stop and report partial output. Other policies need evidence that their continuation or restart behavior is acceptable.
Should every rate limit trigger fallback?
No. Classify the actual condition, remaining budget, compatibility and approved data path. Some failures need configuration or quota resolution rather than another automatic attempt.