Problem Management and Root Cause Analysis

Problem management turns operational evidence into work that reduces recurring or consequential failures. Incident response restores service; problem investigation asks what made the failure possible and which changes would make the next occurrence less likely or less harmful. Both matter, but they have different completion criteria.

Begin with a bounded statement of impact and observed behavior. Preserve the timeline, distinguish facts from hypotheses, and identify conditions that combined to produce the outcome. “Human error” ends an investigation precisely where the useful design questions begin.

TL;DR

Quick Example

This fictional Python gate distinguishes completed implementation from verified corrective action. It checks only the presence of declared fields, using made-up identifiers.

The exact assertions passed locally: implementation marked complete was insufficient, verification with an evidence reference passed, and a missing owner blocked closure. This is not an incident platform, approval mechanism, or proof that any real mitigation works. A record can be fully populated and still contain poor evidence.

Core Concepts

Incident, problem, and change

An incident record describes a disruption and recovery. A problem investigation can connect several incidents or identify a risk before another incident occurs. A change implements a proposed improvement. Keep those links visible without requiring every minor event to become a large investigation.

Contributing causes

A trigger explains what happened at the start; contributing conditions explain why it reached users and why controls did not contain it. A useful causal account connects both to evidence rather than selecting the last person who touched the system.

Learning record

Google’s SRE postmortem guidance describes recording impact, mitigation, causes, and follow-up work, and emphasizes blameless examination of contributing conditions. That approach supports learning; it does not mean avoiding accountable ownership of improvements. SRE postmortem culture

Build a timeline that supports a testable explanation

Start with a precise scope: affected service, operations, users or workload, time range, and observed consequences. Distinguish confirmed impact from estimates. Note which measurements were unavailable and how that uncertainty affects the conclusion.

Record relevant changes, alerts, symptoms, diagnostic steps, decisions, mitigation, and recovery in a common time basis. Clock offsets and delayed logs can reorder events. Preserve original timestamps where needed and explain any reconciliation rather than forcing every record into a convenient sequence.

Label each statement as observation, inference, or unresolved hypothesis. “Errors rose after the release” is an observation about sequence. “The release caused every error” requires additional evidence. Compare affected and unaffected cases, inspect the suspected mechanism, and look for conditions that contradict the hypothesis.

Ask what had to be true for the event to cause harm. Perhaps a worker accepted an oversized input, a memory limit allowed it to consume a shared pool, retries multiplied the load, and an alert arrived after the queue had already filled. Each link suggests a different improvement and a different verification method.

Do not use a fixed number of why questions as a stopping rule. Stop when the explanation is sufficiently supported to choose proportionate actions, and retain open questions when evidence cannot resolve them. An honest uncertainty is more useful than a confident story that nobody can test.

Choose corrective work from the failure mechanism

These are fictional examples, not an organization’s approved control set. Choose actions that address the observed mechanism and can be owned and maintained. “Be more careful” is difficult to verify because it leaves the same system conditions in place.

Separate prevention, containment, detection, and recovery. It may be impossible to eliminate every triggering event, while a bounded queue or safer fallback can reduce its consequences. Explain why the chosen combination is adequate for the risk and what remains possible.

Rank work by consequence, recurrence, uncertainty, effort, and dependencies. A long undifferentiated action list can hide the important change among cosmetic tasks. Give each accepted action a named owner, target date, acceptance method, and link back to the relevant causal claim.

Avoid promising that one completed action prevents every recurrence. The same visible symptom can have several mechanisms, and a future event can combine different conditions. State the narrower outcome the action is expected to improve.

Verify outcomes and preserve the useful record

An implementation review establishes what changed. A verification step establishes whether the change behaves as intended under the relevant condition. Keep those milestones separate so a merged patch does not automatically close a problem.

Use an approved test environment for failure exercises and consequential changes. Reproduce the mechanism with bounded inputs where possible. When reproduction is impractical, identify alternative evidence and its limits: a configuration assertion, a recovery drill, a load test, or a review by the system owner can answer different questions.

Watch the agreed indicators after rollout, but interpret absence of recurrence carefully. A rare event not happening for a week may tell you little. Compare exposure, workload, and detection coverage before treating a quiet period as success.

Retain the explanation, evidence references, implemented actions, verification results, and explicit residual risks in a place operators can find. A known workaround should describe its conditions, owner, and limitations; it should not silently become an unreviewed permanent design.

Review overdue or blocked actions with the responsible owners. If the organization accepts a residual risk instead of implementing a proposed action, record the authorized decision and revisit trigger. The writing of a postmortem is an input to this process, not its finish line.

Best Practices

Keep the investigation proportionate

Use a small record for a bounded issue and a deeper review for major impact or uncertain systemic risk. The required evidence should follow the question being answered.

Make uncertainty visible

Preserve contradictory observations and explain why a hypothesis is favored. Ask what new evidence would change the conclusion.

Review action quality

Require a mechanism, owner, and observable acceptance condition for each important action. Check the resulting behavior after implementation.

Common Mistakes

Bad: The last change is automatically the root cause.

Correct: Use sequence as a clue, then test the causal mechanism against affected and unaffected evidence.

Bad: Blameless means nobody owns follow-up.

Correct: Separate learning from blame while assigning clear responsibility for accepted corrective work.

Bad: The postmortem is approved, so the problem is solved.

Correct: Track implementation, verification, and residual risk through the appropriate operating process.

FAQ

Does every incident require a full RCA?

Use criteria based on impact, recurrence, uncertainty, and learning value. A small incident can need deep analysis, while a well-understood minor event may need a concise record.

Can an investigation have several causes?

Yes. Describe the interacting conditions and their evidence rather than forcing one label to explain a complex outcome.

When can a problem close with risk remaining?

When the organization’s authorized owner explicitly accepts the remaining risk under its process and records the conditions for revisiting it. A script cannot make that decision.

Related Topics

References