Data Incident Response and Reconciliation
A data incident is not resolved merely because a pipeline runs successfully again. Identify the faulty data version, affected consumers and decisions, contain further impact, correct the cause and outputs, and verify the state consumers actually use. Preserve evidence and coordinate response while the investigation continues.
Start with the symptom and the expected contract. Separate a wrong value, missing partition, duplicate record, stale result and unauthorized exposure because their containment and recovery needs differ. Do not assume that the latest successful run replaced every bad copy or reversed every downstream action.
TL;DR
- An incident isn't resolved just because the pipeline runs successfully again.
- Identify the faulty data version and the affected consumers and decisions, then contain, correct, and verify the state consumers actually use.
- Classify the failure — wrong value, missing partition, duplicate, stale result, or unauthorized exposure — because each needs different recovery.
- Preserve evidence before overwriting bad artifacts.
- Reconcile at the key level; matching totals can hide wrong values.
Quick Example
This Python example uses fictional keyed integer outputs. Two records have exchanged values while the total remains unchanged. Correcting the source dictionary still leaves a separate consumer cache wrong.
The totals agree even when keys a and b are wrong. A key-level comparison finds that difference, and the consumer comparison shows it remains after the source correction. Additional checks distinguish a missing key from a present key with a null value. In a real incident, run the same key-level comparison against every consumer-facing copy, not just the corrected source.
Core Concepts
Contain impact and preserve a timeline
Google's SRE incident-response guidance separates coordination, communication and operational work and describes maintaining a shared incident record. Use clear responsibilities so investigating the cause does not leave containment or updates unowned. The exact staffing and authority should fit the organization and incident. Incident-response guidance
Record when the issue was observed, what consumers reported, which releases or data runs are suspected, and what actions were taken. Keep hypotheses distinct from established observations. Avoid overwriting the only faulty artifact before it can be investigated through the approved evidence process.
Containment may mean pausing publication, quarantining an output, stopping an affected downstream operation or serving a clearly identified prior result. Choose the action according to consumer consequences and authority. An old dataset is not automatically a safe substitute if its freshness or interpretation is unsuitable.
Trace run and version identity
Identify source files or snapshots, transformation code/configuration, run identity, produced partitions and publication targets. OpenLineage distinguishes runtime run observations from design-time job or dataset metadata, providing a useful model for what lineage evidence can describe. A declared graph alone does not establish that every actual consumer was observed. OpenLineage object model
Look for copies and derivatives: exports, caches, dashboards, model features, notifications and manual downloads. Record known consumers and explicitly unknown coverage. A successful query for registered dependencies does not prove that an unregistered spreadsheet or external integration never used the bad output.
This is an editorial incident map, not a completed investigation or permission to alter production data.
Reconcile more than the grand total
Choose comparisons that match the failure mode. Counts and sums can reveal broad losses or duplication, while keyed values, partition boundaries and domain relationships can expose errors those aggregates miss. The example demonstrates that equal sums can hide a swap.
Define keys, null handling, units, period, rounding and intentional revisions before declaring equality. A missing row and a present null are different states. Some domains require tolerance for numerical representation; others require exact identity. The reconciliation rule must follow the data contract rather than whichever comparison passes most easily.
Correction and publication need a version policy
Decide whether recovery replaces an existing output, publishes a corrected revision or produces a compensating artifact. Preserve enough provenance to explain what consumers should use and why. Do not silently edit historical results when the domain or audit process requires an explicit revision.
A replay must use the intended code and inputs and avoid repeating unintended effects. Data recomputation, notification delivery and external system updates may need different recovery paths. Restarting the complete pipeline can create a second problem if it reissues actions that already occurred.
Best Practices
Create one response record linking the symptom, expected contract, affected identities, containment, owners, hypotheses, decisions and verification. Include the consumer-facing impact at the level supported by evidence. Avoid declaring all users unaffected merely because the visible technical error is small.
Build a minimal failing fixture from authorized evidence, with sensitive details removed or controlled appropriately. Verify that the correction addresses the cause and relevant adjacent cases. Keep the original failure observable in tests rather than changing the expected result to match a rerun.
Reconcile at several meaningful boundaries: source acceptance, transformed output, published partition and consuming application. State which differences are expected due to deduplication, aggregation or correction. A raw count mismatch is not always an error, but an unexplained one remains an open question.
Verify actual downstream recovery. Refresh or rebuild affected copies through their authorized procedures, confirm version identity and test the consumer's original task. The example's stale cache illustrates why a corrected source alone is insufficient evidence.
Track actions that cannot be reversed by replacing data. An external message, export or consequential decision may need a separate owner and recovery process. Involve the applicable business, security, privacy or legal specialists when the actual incident affects their responsibilities. This generic example contains no personal data or real decisions.
After recovery, record contributing causes, detection gaps and concrete follow-up ownership. Distinguish restoring service from preventing recurrence. A useful post-incident improvement can be a clearer publication contract or lineage boundary rather than another dashboard with no response owner.
Reverse a closure decision when a consumer still reads a bad version, reconciliation has an unexplained gap or important impact coverage remains unknown. Keep the unresolved scope explicit and assign the next check. A green pipeline is evidence about orchestration, not an automatic certificate of data recovery.
Common Mistakes
Equating a successful rerun with recovery
Checking only aggregate totals
Overwriting the evidence
Claiming complete impact from partial lineage
FAQ
Do matching totals prove reconciliation?
No. The example's swapped values preserve the sum. Choose keyed, aggregate and domain checks that match the failure and expected contract.
Does correcting the source fix consumers automatically?
Not necessarily. Caches, exports and derived results can retain the old data. Verify the actual consumer copy and original workflow.
How do I know a data incident is actually resolved?
When consumer-facing copies match the corrected contract at the key level, downstream actions taken on the bad data have been reviewed or reversed, and the owners of affected consumers agree. A successful rerun is one input to that decision, not the decision itself.