Data Contracts & Schema Evolution for Data Pipelines

Data Contracts and Schema Evolution

Most broken dashboards and silently wrong reports start upstream, with a change that made perfect sense to the team that made it. An application team renames a column during a refactor. A service starts sending amounts in dollars instead of cents. An order table that had one row per order now has one row per order line. Nothing errors — or the pipeline fails at 2 a.m. — and the analytics, finance, and machine-learning teams downstream discover the change days later, from the numbers.

A data contract is an explicit agreement between the producer of a dataset and its consumers: what the data looks like, what it means, how fresh and complete it will be, who owns it, and how it is allowed to change. Schema evolution is the discipline of changing data under that agreement without breaking the people who depend on it. Together they turn data interfaces into something as deliberate as APIs.

TL;DR

A data contract covers schema, semantics, quality rules, service levels, ownership, and change policy — a schema alone is not enough.
Publish interfaces, not internal tables: consumers should depend on a contracted dataset or event, not on whatever the application's database looks like today.
Classify changes by compatibility; semantic changes — units, meaning, grain — are the most dangerous because nothing errors.
Enforce contracts automatically: in producer CI, schema registries, ingestion validation, and transformation tools such as dbt.
Ship breaking changes as a new version with an overlap period, using lineage to find and notify consumers.
Start with the few datasets that matter most — billing, finance, core product metrics — rather than contracting everything.

Quick Example

A contract for an orders dataset, kept in version control next to the code that produces it:

The same file documents the dataset for consumers, drives automated checks for the producer, and tells everyone what kind of change requires a new version.

Core Concepts

What a Contract Contains

A schema describes shape. Most damaging changes keep the shape and change the meaning, which is why semantics, grain, and units belong in the contract.

Where Contracts Sit

Contracts belong at the boundaries where ownership changes hands: between an application team and the data platform, between the data platform and its consumers, and on every event stream other teams read.

Interfaces, Not Internal Tables

Change data capture makes it easy to replicate every application table into the warehouse — and then every column becomes something someone depends on, so the application team can no longer refactor its own database. Publish a deliberate interface instead: a contracted view or table, or events written through an outbox, so internal schemas can change while the contract stays stable.

Compatibility Rules for Tables

Event formats have well-defined compatibility modes, covered in Event Schema Evolution. Tables consumed by SQL need equivalent rules:

Enforcing Contracts

A contract that only lives in a document is a hope. Enforce it at several points:

In dbt, an enforced model contract fails the build when the model's columns or types differ from the declaration:

Which constraints a warehouse actually enforces varies by platform — many enforce only not_null — so pair contracts with data tests. See dbt and Data Quality Management. Tools such as the Data Contract CLI can lint contract files, test a dataset against its contract, and report breaking changes between versions.

Evolving a Contract

Breaking changes are sometimes necessary. Handle them like a versioned API:

Propose the change with the reason and the new contract version.
Find consumers through lineage and the contract's consumer list, and notify them.
Publish the new version alongside the old one — a new table, view, or topic — and populate both.
Migrate consumers, tracking who has moved.
Deprecate the old version with a date, and monitor for remaining readers.
Remove it once usage has stopped.

This is the expand-and-contract pattern used for application databases — see Release Rollback and Database Compatibility — applied across team boundaries. Semantic versions make the kind of change obvious: a major version is breaking; minor and patch versions are not.

Ownership

Contracts work when responsibilities are clear:

The producer owns the contract, keeps the data within it, and runs the change process.
Consumers register their use, so they are notified, and avoid depending on anything outside the contract.
A platform team provides tooling: contract formats, CI checks, registries, lineage, and monitoring.
Data owners resolve disputes about priorities and acceptable use.

The idea is central to data mesh, where every dataset offered to others is a product with an owner and a published contract. The practice is useful without adopting data mesh as a whole.

Best Practices

Describe Semantics Explicitly

Units, currencies, time zones, grain, and definitions such as "active customer" belong in the contract. Most costly data incidents are misunderstandings about meaning, not type errors.

Keep Contracts Next to the Producing Code

When the contract lives in the producer's repository, the same pull request that changes the data must change the contract, and the same CI run checks both.

Quarantine, Don't Silently Drop

Records that violate a contract should go to a quarantine area with the reason recorded, and the producer should be alerted. Silently dropping them hides problems; silently loading them spreads them.

Avoid SELECT * Downstream

Explicit column lists in staging models keep new upstream columns — including new personal data — from flowing everywhere by accident, and make dependencies visible.

Common Mistakes

Replicating Internal Tables as the Interface

A Contract Nobody Enforces

Silent Semantic Changes

Breaking Changes Without an Overlap

FAQ

What is a data contract?

An agreement between the producer of a dataset and its consumers covering structure, meaning, quality, freshness, ownership, and how the data may change. It is usually written as a machine-readable file and enforced by automated checks, so the agreement is kept rather than merely documented.

How is a data contract different from a schema?

A schema describes structure: field names and types. A contract includes the schema plus semantics, quality rules, service levels, classification, ownership, and change policy. Many breaking changes, such as a unit change, keep the schema identical.

Who owns a data contract?

The producing team. They control the data and must be the ones to keep it within the contract and manage changes. Consumers participate by registering their use and reviewing proposed breaking changes.

Do we need a standard contract format?

It helps but is not essential. Open standards such as the Open Data Contract Standard give tools a common format; a simple, consistent YAML file checked in CI delivers most of the value. Enforcement matters more than the format.

Which datasets should get contracts first?

Those whose failure is most expensive: billing and revenue data, financial reporting inputs, core product metrics, and datasets feeding machine-learning models in production. Expand as teams see the benefit.