Reliability Testing & Dependency Failure Drills

Reliability Testing and Dependency Failures

Every service depends on others — databases, caches, payment providers, identity services, internal APIs — and each of them will eventually be slow, return errors, or disappear. Whether that becomes a minor blip or a full outage depends on code paths that rarely run: timeouts, retries, circuit breakers, fallbacks, and failover. Untested, these paths are where surprises live: a timeout longer than the caller's, retries that triple load on a struggling dependency, a "fallback" that throws its own exception, a replica promotion nobody has performed since the cluster was built.

Reliability testing exercises those paths deliberately — in tests, in staging, and carefully in production — so the system's real behavior under failure is known before customers discover it.

TL;DR

Build a dependency failure matrix: for each dependency and failure mode, write down the expected behavior.
Inject faults — latency, errors, connection resets, full outages — with proxies, meshes, or chaos tools.
Verify timeouts fit within caller budgets, retries are bounded, circuit breakers open, and fallbacks work.
Define and test degraded modes: what the product does when non-critical features are unavailable.
Rehearse failover of databases, zones, and regions, and measure recovery time.
Run game days and chaos experiments with a steady-state hypothesis, a limited blast radius, and abort conditions.

Quick Example

An integration test that slows a dependency with Toxiproxy and checks that the product page degrades gracefully within its latency budget:

The test fails if someone removes the timeout, sets it longer than the page's budget, or replaces the fallback with an exception.

Core Concepts

The Dependency Failure Matrix

Writing the matrix forces decisions that are otherwise made accidentally during incidents.

Failure Modes to Inject

Resilience Mechanisms Under Test

Timeouts: every remote call has one, and each is shorter than the caller's remaining budget.
Retries: only for idempotent operations, with exponential backoff, jitter, and retry budgets.
Circuit breakers: stop calling a failing dependency and recover after a probe succeeds. See Circuit Breakers.
Bulkheads: separate connection pools or thread pools so one slow dependency cannot exhaust all resources.
Fallbacks and degraded modes: cached data, defaults, hidden features, or queued work instead of errors.
Load shedding: rejecting excess work early during overload.

Levels of Reliability Testing

Running Chaos Experiments Safely

Steady-state hypothesis: define normal using SLO metrics — success rate, latency, throughput.
Hypothesis: "If the cache cluster fails, success rate stays above 99.5% and p99 latency below 800 ms."
Blast radius: one instance, one zone, or a small percentage of traffic first.
Abort conditions: automatic rollback when SLO burn exceeds a threshold.
Observe and record: did alerts fire, did dashboards show the problem, did runbooks work?
Fix and repeat: turn findings into code changes, tests, and automated experiments.

Tools include AWS Fault Injection Service, Azure Chaos Studio, Chaos Mesh, LitmusChaos, Gremlin, and service mesh fault injection. See Chaos Engineering.

Failover Drills

Failover mechanisms decay silently: credentials expire, DNS TTLs change, replicas fall behind, runbooks drift. Drill them on a schedule — database primary failover, cache cluster replacement, zone evacuation, region failover — and record time to detect, time to fail over, data loss (if any), and manual steps that should be automated. See Disaster Recovery Planning.

Best Practices

Start in Tests, Then Move Outward

Automated fault injection in CI catches regressions cheaply; staging and production experiments validate the parts tests cannot simulate.

Tie Experiments to SLOs

Use SLO metrics for steady state and abort conditions, so experiments measure what users experience. See SLOs.

Test Observability Too

Every experiment should check that alerts fired, dashboards showed the fault, and on-call engineers could find the cause.

Make Findings Permanent

Each discovered weakness becomes a fix plus a regression test or automated experiment, so it stays fixed.

Common Mistakes

Timeouts Longer Than the Caller's

Retries That Amplify Outages

Fallbacks Nobody Has Run

FAQ

What is reliability testing?

Deliberately testing how a system behaves when components fail — slow or failing dependencies, lost instances, failovers, and overload — to verify that timeouts, retries, circuit breakers, fallbacks, and recovery procedures work as intended.

How is chaos engineering different from reliability testing?

Chaos engineering is a disciplined form of reliability testing: running controlled experiments, often in production, with a steady-state hypothesis and limited blast radius. Reliability testing also includes automated tests in CI, load tests, and failover drills.

Should I run failure experiments in production?

Eventually, for the scenarios staging cannot reproduce — but only after testing in lower environments, with small blast radius, SLO-based abort conditions, and the team watching.

How do I test a dependency outage locally?

Run the dependency behind a fault-injecting proxy such as Toxiproxy in containers, or replace its client with a fake that can be configured to time out, error, or refuse connections.

What is a degraded mode?

A planned way for the product to keep working with reduced functionality when a dependency fails — showing cached data, hiding non-essential features, or queuing work for later instead of returning errors.