Flaky Test Diagnosis
A flaky test produces inconsistent outcomes without an intended change to the behavior being tested. Diagnose the uncontrolled condition: shared state, execution order, asynchronous timing, dependency behavior, resource contention, or an assertion that does not match the contract.
Preserve the first failure before rerunning. A passing retry is evidence about intermittency, not proof that the original failure was harmless. Users can encounter the same race once and have no retry button.
TL;DR
- Retain first-failure evidence, environment details, and the exact test order or seed.
- Separate a faulty test assumption from an intermittent application defect.
- Reproduce by controlling one timing, order, state, or dependency variable at a time.
- Wait for contract-relevant conditions with bounded timeouts.
- Verify the repair against the original reproducer and neighboring cases.
Quick Example
This complete Python example deliberately reproduces an order dependency in a fictional in-memory test fixture, then checks isolated state. It does not run pytest or a browser and uses no random timing.
The negative control is deterministic: one order fails, the other passes. The repair changes fixture ownership rather than repeating the assertion until it becomes green. In a real suite, reproduce against the actual storage, process, or session boundary involved.
Core Concepts
Uncontrolled state
Tests depend on more than their source code: environment variables, files, database rows, clock, random generator, browser storage, service state, and preceding tests can affect outcomes. Pytest’s flaky-test guidance highlights insufficient isolation and order dependence as common causes. Pytest diagnosis
Timing and conditions
A sleep assumes completion by a chosen time. A bounded condition wait observes the expected state or fails with a useful timeout. Choose the condition carefully: an element being visible does not mean a backend write is complete.
Test defect or product race
If the product promises a result before returning success, a test that sometimes observes missing data may expose a real defect. Do not weaken the assertion until the test no longer checks the promise. First establish the intended contract and the evidence around its boundary.
Turn intermittency into a controlled experiment
Create a compact failure record: test identifier, first error, assertion, trace or relevant logs, source revision, environment, worker count, order/seed, dependency versions, and resource conditions. Redact credentials and private records without removing the causal timestamps and identifiers.
Change one factor at a time where possible. If isolation fixes the symptom, identify the actual shared resource before declaring victory. A serial run can hide a race that real concurrent users still trigger.
For order problems, preserve a failing sequence and minimize it. If a randomized run finds the issue, retain the seed and resulting order; a seed without the runner/environment details may not reconstruct the same execution. Check cleanup on failed setup and cancellation, not only successful test completion.
For network problems, separate the application’s timeout/retry behavior from external service availability. Use deterministic fault-focused tests for the client rule and a smaller real integration check for the dependency. Avoid stubbing the dependency everywhere and losing the boundary that matters.
Repair the cause and retain the signal
Common repairs include per-test data ownership, local temporary directories, independent browser contexts, deterministic clocks, explicit event/condition synchronization, and cleanup in a guaranteed lifecycle hook. Choose the change that restores the intended contract with the smallest new assumption.
Playwright recommends isolated tests and web-first assertions that wait for observable conditions. Its retry reporting distinguishes a first-run pass, a pass after failure, and an exhausted failure. Keep those categories visible instead of presenting only the final successful attempt. Testing practices · Retry reporting
After the repair, rerun the exact reproducer and relevant neighboring conditions: both orders, parallel execution where supported, failure cleanup, and the original dependency timing. Define a meaningful observation window or repetition plan from the failure frequency and cost. No finite number of clean runs proves a probabilistic defect impossible, so retain the regression and monitoring.
If quarantine is necessary to unblock unrelated work, give it an owner, a reason, a tracking item, and a review date. Preserve coverage through another suitable check where possible. Quarantine changes release evidence; it does not complete the repair. Repeatedly extending a timeout or marking a test optional without a diagnosis turns an intermittent signal into missing coverage.
Best Practices
Keep first-failure artifacts
Store enough evidence to reconstruct the failed attempt before cleanup. Successful retry logs alone can remove the only useful clue.
Assert the real contract
Use waiting and tolerances only where the product permits eventual or approximate behavior. Do not convert a required immediate guarantee into eventual success because the original assertion was inconvenient.
Review ownership of shared resources
Assign unique test/run identifiers and scoped cleanup for records, queues, files, and sessions. Verify failed tests cannot leave state that poisons the next run.
Measure reliability of the suite
Track flaky outcomes, affected journeys, diagnosis time, and unresolved quarantines. A high pass rate can conceal a small set of important journeys that repeatedly fail.
Common Mistakes
Bad: Calling a successful retry a fixed test.
Correct: Retain the first failure and investigate the condition that changed between attempts.
Bad: Increasing every timeout when CI is slow.
Correct: Measure the delayed boundary and use a bounded meaningful condition; check actual resource contention.
Bad: Disabling parallelism as the final repair.
Correct: Determine whether shared state is invalid or the test models a real concurrency requirement, then fix the appropriate boundary.
FAQ
Can a flaky test reveal a real application bug?
Yes. Intermittent behavior may violate the user-facing contract. Confirm that contract before weakening assertions or replacing dependencies.
Should tests never use timeouts?
Timeouts bound waiting. The problem is using an arbitrary delay as proof of completion rather than waiting for a relevant condition.
How many reruns prove a repair?
There is no universal number. Use the original reproducer, relevant variable changes, observed frequency, and ongoing regression evidence.
When is quarantine acceptable?
When the coverage consequence is understood and the issue has an owner and review path. It is temporary release-risk management, not a completed diagnosis.
Related Topics
- Testing & Quality Assurance — Maintain a useful test portfolio.
- End-to-End Testing Strategy — Choose durable integration journeys.
- Concurrency and Parallel Computing — Understand timing and shared-state interactions.