Great Expectations: Assertions and Pipeline Decisions

Great Expectations

Great Expectations expresses assertions about data and evaluates them against a chosen batch. Begin with a meaningful data contract: which records are included, what must hold, which exceptions are permitted and who acts when the contract fails.

A validation result describes the tested conditions. It does not automatically repair data, stop every downstream job or establish overall fitness for use. Connect the result to an explicit pipeline decision and retain enough context for an owner to diagnose the failure.

TL;DR

Define the batch and business meaning before selecting Expectations.
Test required values separately from range or distribution rules.
Include passing and deliberately failing batches so a green run is meaningful.
Distinguish returned failure, evaluation error, notification and pipeline enforcement.
Make reports actionable while limiting sensitive failed-row data and access.

Quick Example

This complete example targets GX Core 1.24.1 with pandas. It uses an ephemeral context with explicit in-memory stores and disables optional analytics for this fixture. All counts are fictional; no database, cloud service or external file is used. Install compatible dependencies in an isolated environment before running it.

Expected output reports checks for valid, out-of-range and missing-value batches. The example targets GX Core 1.x with pandas. The example evaluates individual Expectations; it does not run a Checkpoint, publish Data Docs, send an alert or enforce a production pipeline stop. The not-null rule is deliberate: a range check alone should not be assumed to establish requiredness.

Core Concepts

Data Context, asset and batch

A Data Context provides access to GX configuration and validation components. A Data Source and Data Asset describe how data is organized for GX, and a Batch Definition selects the records to validate at runtime. The example passes one in-memory pandas DataFrame as a whole batch. DataFrame connection

Expectations and Suites

An Expectation is a verifiable assertion; an Expectation Suite groups assertions. Choose rules from the actual consumer contract rather than adding every available check. A column’s allowed range, requiredness, type and uniqueness can express different concerns. GX Core overview

Validation and response

A Validation Definition associates batch selection with a Suite. Checkpoints run Validation Definitions and can execute configured Actions based on results. Sending a message and preventing bad data from reaching a consumer are separate integration behaviors. Checkpoint Actions

Build rules around a consumer decision

Start with the consequence of a failure. A missing identifier might prevent an operation entirely; an unusual distribution might warrant investigation without blocking every record. State that distinction with the responsible data and product owners before choosing a threshold.

Keep thresholds and exceptions versioned with their rationale. A tolerance chosen solely to make the current batch pass can hide a real change. If a business rule allows a fraction of exceptions, record why that fraction is acceptable and how affected records are handled.

Check missing values explicitly. A value-range assertion and a requiredness assertion need not have identical missing-data semantics. Use the actual Expectation documentation and include a missing-value fixture, as this example does. Include an empty-batch case in the production contract instead of assuming a rule over zero records establishes successful ingestion.

Prefer a small set of consequential assertions first. Expensive redundant checks can slow a pipeline and create noisy failures without improving the consumer decision. Add a rule when its result has a clear interpretation and owner.

Join results to a failure policy and useful report

Decide whether each failure blocks publication, quarantines a bounded batch, degrades an optional output or creates a nonblocking investigation. Define where that decision runs relative to downstream writes. A check executed after publishing bad data is an observation, not a prevention gate.

Distinguish data failing an assertion from validation failing to execute. A connection error, malformed rule or unavailable dependency should not silently become a passing result. Test the error path and how retries affect work already completed.

A useful report identifies dataset/batch, input identity or time window, rule/Suite version, expected condition, observed result, affected count and responsible owner. Include next action, exception status and rerun identity. Preserve enough information to distinguish a changed source from a changed test.

GX’s Checkpoint Actions can update Data Docs or trigger configured responses; the chosen Action and severity settings determine what happens. Verify the configured response, such as a Data Docs update or a notification, in an authorized test before relying on it. Actions and severity

Avoid publishing raw failed-row values indiscriminately. Validation results and generated reports can expose sensitive data. Choose result verbosity, redaction, access and retention according to the actual dataset. A report should help an authorized owner diagnose the problem without creating unnecessary copies.

For recovery, preserve the failing input or an appropriate controlled reference, fix the cause, rerun the relevant rules and verify that downstream processing resumes without duplication. Reconsider the rule if a documented contract changed; do not suppress the failure merely because an alert is inconvenient.

Best Practices

Test a failure before trusting a pass

Keep valid, missing and invalid examples with the rule. Verify both returned results and the pipeline’s chosen action.

Record what was validated

Retain batch identity, selection parameters and rule version. A successful validation of the wrong records answers the wrong question.

Keep integration evidence separate

Check stop/quarantine behavior, retry, report access and notification delivery explicitly. A local assertion does not demonstrate those systems.

Common Mistakes

Bad: Assuming any failed Expectation automatically stops a job.

Correct: Connect results and errors to an explicit pipeline action at the correct point.

Bad: Treating a range rule as proof that no values are missing.

Correct: Add and test the requiredness condition separately.

Bad: Exporting complete failed rows into every report.

Correct: Use the minimum diagnostic detail with appropriate access, redaction and retention.

FAQ

Does a passing Suite prove the dataset is correct?

It supports the assertions evaluated over the selected records. Missing rules, wrong selection and untested business meaning can still leave important problems.

Does this example send notifications?

No. It runs individual in-memory validations and checks their success values. Checkpoints, Actions and external delivery remain unexecuted.

Should every anomaly block the pipeline?

Choose by consequence and the consumer contract. Define quarantine, exception and recovery behavior rather than turning every unusual value into the same action.