Chaos Engineering
Chaos engineering is the practice of deliberately injecting failures into a system to verify it survives them — killing instances, adding network latency, filling disks — as controlled experiments rather than waiting for production to run the experiment for you at 3 a.m.
The discipline was born at Netflix (Chaos Monkey randomly terminated production instances, forcing every service to tolerate instance death), but the modern practice is less monkey, more method: hypothesis-driven experiments with bounded blast radius. Your architecture is full of resilience claims — "we fail over automatically," "retries handle that," "the cache degrades gracefully." Chaos engineering is how those claims become tested facts before an outage tests them for you.
TL;DR
- Chaos engineering = experiments, not vandalism: define steady state → hypothesize it survives a failure → inject the failure with a minimal blast radius → measure → widen or fix.
- You're testing the system's response — timeouts, retries, failover, alerting, runbooks, humans — not whether a pod can die (it can).
- The classic injections: instance/pod kill, network latency and partition, dependency failure, resource exhaustion, AZ/region loss, expired certs, DNS failure.
- Prerequisites first: without observability and a steady-state metric, you're injecting failure blind — that's not an experiment, it's an incident with extra steps.
- Start in staging, then production with guardrails (abort conditions, off-hours, 1% scope) — production is the only place the real dependency graph exists.
- GameDays — scheduled team exercises running experiments together — are the highest-value entry point and test the humans and runbooks too.
The Method
A chaos experiment has the same shape as any experiment:
Two rules carry the safety weight:
- Blast radius: start with the least you can learn from — one instance, one canary user segment, 1% of traffic — and expand only after surviving smaller scopes.
- Abort conditions: a big red button plus automatic stop rules ("halt if error rate exceeds X") defined before injection. Experiments must be reversible in seconds.
What to Inject
The pattern across all of them: the failure is trivial; the response is the system under test. Latency injection is the highest-yield starter — real dependencies rarely die cleanly, they get slow, and slow dependencies expose missing timeouts better than any code review.
Tooling
Two adjacent practices worth knowing: continuous chaos (scheduled randomized experiments as regression tests for resilience, à la Chaos Monkey) and chaos in CI — Toxiproxy-style dependency faults in integration suites, so "handles DB timeout" is a unit-testable property, not a production discovery.
GameDays: The Human Half
Most outages are prolonged not by the failure but by the response — unclear ownership, stale runbooks, dashboards nobody can find. A GameDay exercises exactly that:
- Pick a scenario ("primary database failover") and a date; invite the owning team plus on-call.
- Someone runs the injection; the team responds as if real — using the actual runbooks, dashboards, and escalation paths.
- Observe: Did alerts fire before humans noticed? Was the runbook accurate? Who knew what to do?
- Debrief like a postmortem: action items for every gap in tooling, docs, and knowledge.
GameDays deliver most of chaos engineering's value with the least tooling — a great first step for teams not ready to automate injections, and the only way to test the sociotechnical system (pages → humans → decisions) that actually resolves incidents.
Common Mistakes
Chaos Without Observability
If you can't measure steady state, you can't run an experiment — you can only cause mysteries. Metrics, alerting, and dashboards for the target service are prerequisites, not nice-to-haves.
Starting in Production, Unbounded
The Chaos Monkey legend makes teams skip the ladder. The ladder exists: local/CI (Toxiproxy) → staging → production canary scope → production meaningful scope. Skipping rungs converts experiments into self-inflicted SEV-1s — which sets the program's reputation back a year.
Testing Only Infrastructure
Pod-kill experiments pass because Kubernetes reschedules pods — congratulations, you tested Kubernetes. The interesting hypotheses live at the application layer (fallbacks, timeouts, degradation) and human layer (alerts, runbooks).
No Fix Loop
Finding the same missing timeout in three consecutive GameDays means the program produces reports, not resilience. Every failed hypothesis becomes a tracked engineering item plus a repeatable regression experiment.
Chaos as a Team of One
An enthusiast running experiments against other teams' services without their engagement generates resentment and pager fatigue. Chaos is done with service owners — ideally by them — with the platform team providing tools and guardrails.
Ignoring Error Budgets
Experiments spend real error budget when they degrade users. Schedule against budget health, and count experiment burn honestly — a team out of budget shouldn't be running production chaos that week.
FAQ
Isn't this just testing? How is it different from integration tests?
Tests verify known behavior against specifications in controlled environments. Chaos experiments probe emergent behavior of the whole sociotechnical system — real dependency graphs, real timeouts interacting, real humans responding — where the interesting failures are precisely the ones nobody specified. They complement, not replace: promote each chaos finding into a deterministic test where possible.
Do we have to do it in production?
The honest answer: staging finds the cheap 60%; production finds the truth, because only production has the real traffic patterns, data volumes, and dependency graph. Mature programs earn their way to production with guardrails. Teams that never get there still benefit — just calibrate confidence accordingly.
When is a team ready for chaos engineering?
Checklist: solid observability on the target service, on-call and incident process that functions, known steady-state metrics, and leadership buy-in for "we broke it on purpose." If routine deploys still cause regular incidents, fix that first — chaos engineering hardens systems that basically work; it doesn't rescue ones that don't.
How does this relate to SRE and error budgets?
Chaos engineering is SRE's experimental arm: SLOs define what "resilient enough" means, error budgets bound how much experiments may spend, and postmortem culture is where findings become fixes. Netflix-style resilience without SRE-style measurement tends to drift into theater.
What's a good first experiment?
Add latency (not failure) to one non-critical dependency of one service in staging, with a 5-minute duration and the owning team watching dashboards. It's low-risk, and it almost always finds a missing timeout or an alert that doesn't fire — instant, safe proof of value.
Related Topics
- Site Reliability Engineering — The framework chaos experiments serve
- SLOs — Steady state and the budget experiments spend
- High Availability — The claims being verified
- Monitoring — The prerequisite sensory system
- Kubernetes — Where Chaos Mesh/Litmus operate
- Performance Testing — Failure under load
- Microservices — The architecture whose emergent failures need probing