Postmortems: Blameless Incident Review That Works

Postmortems & Incident Review

A postmortem exists to convert an outage into organizational learning. That's a lower bar than it sounds and most organizations miss it, producing documents that identify a root cause ("human error"), assign a corrective action ("be more careful"), and file the result somewhere nobody reads.

The failure is almost always cultural rather than procedural. If describing what you did during an incident can hurt you, you will describe it carefully — and the careful version omits exactly the details that would have taught everyone something. Blamelessness is not kindness; it's the precondition for getting accurate information. An engineer who ran the wrong command knows why it seemed like the right command at the time, and that reasoning is the finding. You only get it if it's safe to say.

The second recurring failure is the action item nobody does. A postmortem that produces twelve unowned tasks has produced nothing, and the same incident recurs eight months later with a new document attached.

TL;DR

Blameless means analyzing systems and decisions, not assigning fault. It's how you get true information.
People act reasonably given what they knew at the time. Hindsight makes the wrong choice look obvious.
Build a timeline first, in UTC, from logs and messages — before any analysis.
Prefer contributing factors to a single "root cause." Real incidents have several.
Ask "how could the system have prevented or caught this?" not "who did it?"
Action items need an owner, a date, and a ticket. Three real ones beat twelve aspirational ones.
Track time to detect and time to mitigate separately — they have different fixes.
Publish widely. A postmortem read only by the team that had the incident teaches one team.

Quick Example

Three properties make that document useful: the timeline is factual and separates detect from mitigate, the contributing factors are plural and structural, and every action has a name and a ticket. The "what we are NOT doing" section is worth stealing — it prevents the overcorrection that gets proposed in every incident review.

Core Concepts

Blamelessness

The principle underneath is that people act rationally given the information, time pressure, and tooling available to them. If someone made a decision that looks obviously wrong in hindsight, the question is what made it look right at the time — and the answer is usually about the system: a misleading dashboard, an ambiguous flag name, a runbook that said something different, a deploy tool that made the dangerous option the default.

Blameless does not mean consequence-free. Malice and repeated recklessness are performance matters, handled separately by a manager. Conflating the two — using postmortems as a performance instrument — destroys the process permanently, and everyone can tell when it's happening.

Contributing factors, not root cause

Real incidents don't have one cause. They have several conditions that individually were survivable and collectively weren't — this is the Swiss cheese model, and it holds up well in practice.

Stopping at a single root cause produces a single narrow fix. Enumerating contributing factors gives you several places to intervene, and lets you choose the cheapest effective ones.

The "five whys" technique tends to produce a single causal chain and can encourage stopping at a person. Asking "what would have prevented this?" and "what would have caught this sooner?" produces better material.

Detect vs. mitigate

These have completely different remedies and should be measured separately. A 40-minute incident that took 35 minutes to detect is a monitoring problem; one that took 35 minutes to mitigate after instant detection is a tooling or runbook problem. Reporting only total duration hides which.

Action items that get done

Every action needs an owner (a person, not a team), a date, and a ticket in the normal backlog. Actions tracked only in a postmortem document don't get done, because nothing surfaces them at planning time.

Three or four real actions beat twelve aspirational ones. Prioritize by which factor, if removed, would most reduce the impact of a recurrence — and be explicit about what you're choosing not to do and why.

The meeting

The facilitator not having been a responder matters. It's very hard to run a neutral review of an incident you were in the middle of, and a facilitator with skin in the game will unconsciously steer toward or away from certain factors.

Best Practices

Write the timeline before the analysis

Facts first, from logs, chat transcripts, deploy history, and alert records. Analysis that starts before the timeline is complete tends to construct a narrative and then find evidence for it. Use UTC — an incident that spans time zones is confusing enough already.

State the blameless framing at the start of every review

Not once when you adopt the practice. Every meeting, out loud. It's a reminder for the room and a signal to anyone whose actions are about to be discussed, and its absence is felt immediately.

Include what went well

The escalation that worked, the runbook that helped, the good judgment call under pressure. Practices that work are worth reinforcing and spreading, and a document that's purely a catalogue of failures is one people dread producing.

Publish widely and make them searchable

A postmortem read only by the team involved teaches one team. Published in a searchable place, it teaches the organization — and it's the material a new engineer reads to understand how the system actually fails. Redact customer data, not the reasoning.

Track action items in the normal backlog

With the same tickets, the same prioritization, and the same visibility as feature work. Review open postmortem actions monthly at a leadership level. Unfinished actions are the mechanism by which the same incident recurs.

Time-box the process to the severity

A SEV-1 warrants a full written postmortem and a review meeting. A minor degradation warrants a short written note. Requiring the full treatment for everything means it stops happening for anything.

Look for patterns across incidents

Five postmortems in six months all mentioning "monitoring didn't catch it" is a bigger finding than any one of them. Review the corpus quarterly and look for repeated contributing factors — those are where systemic investment pays.

Separate the review from performance management

If a postmortem can affect someone's rating, it's not blameless regardless of what you call it. Performance concerns go through the manager, in one-on-ones, on a separate track. Any leak between the two channels ends the honest reporting immediately and permanently.

Common Mistakes

Blaming a person

Stopping at one root cause

Action items with no owner or date

Only doing postmortems for major outages

Writing them and not reading them

Overcorrecting

FAQ

Isn't someone sometimes actually at fault?

Occasionally, and that's handled separately. Malice or repeated recklessness is a performance matter for a manager in a private conversation. The overwhelming majority of incidents involve a competent person making a reasonable decision with incomplete information in a system that permitted the mistake — and treating those as personal failures buys you nothing and costs you all future honesty.

How soon after the incident?

Timeline drafted within 24 hours while memories are fresh; review meeting within three to five days. Immediately after is too raw and the data isn't assembled. More than a week and details blur and urgency evaporates. For a long incident, start the timeline during it — a shared incident channel with timestamps does most of this automatically.

Who should write it?

Someone who was involved, usually the incident commander or the service owner, because they have the context. A facilitator who was not involved should run the review meeting. Splitting these keeps the document informed and the discussion neutral.

Should postmortems be public to customers?

For incidents with significant customer impact, a public version builds far more trust than silence. Include what happened, the impact, and what you're doing about it; omit internal names, specific infrastructure detail, and anything with security implications. Companies with a strong reputation for reliability generally publish more, not less.

How do we make people actually do the action items?

Put them in the normal backlog with normal tickets, give each one a named owner and a date, and review open items monthly at a level that has authority to prioritize. Also: fewer of them. Three actions that get done beat twelve that don't, and a postmortem with twelve actions is usually a postmortem where nobody prioritized.

What about near misses?

Review them, and they're often the best value in the whole practice — you get the lesson without the outage. A deploy caught in staging, an alert that fired before impact, a mistake noticed in review: all of these reveal the same system weaknesses at zero cost. Teams that only review actual outages are declining free information.