Change Data Capture (CDC): Debezium & Log-Based Streaming

Q: Can I use CDC with a managed database?

Yes, with a setup step. RDS/Aurora Postgres need rds.logical_replication = 1; Cloud SQL has a logical decoding flag; Azure Database for PostgreSQL exposes wal_level. MySQL variants need row-based binlog. A few serverless offerings restrict replication slots — check before designing around it.

Change Data Capture (CDC)

Change data capture turns a database into a stream. Instead of polling tables for rows that changed, CDC reads the database's own replication log — the Postgres WAL, MySQL binlog, MongoDB oplog — and emits an ordered event for every insert, update, and delete, typically within milliseconds of the commit.

That one capability quietly solves a long list of problems: keeping a search index in sync, feeding a warehouse without nightly batch jobs, invalidating caches precisely, replicating between heterogeneous databases, and letting services react to data changes without the writing service knowing they exist. It also removes the need for the dual write — the pattern where an application writes to its database and then publishes an event, and loses one of them whenever the process dies in between.

TL;DR

CDC streams row-level changes out of a database by reading its replication log, not by polling.
Log-based CDC is the real thing: complete (captures deletes), low-latency, and near-zero load on the source. Query- and trigger-based CDC are compromises.
Debezium is the de facto open-source implementation, usually running on Kafka Connect and writing to Kafka.
A connector starts with a snapshot of existing rows, then switches to streaming — consumers must handle both.
Delivery is at-least-once, so every consumer needs to be idempotent.
Use the transactional outbox when you want to publish meaningful domain events rather than raw table rows.
Watch the replication slot: a stalled consumer makes the source database retain WAL until the disk fills.

Quick Example

Postgres → Kafka with Debezium, then a consumer that keeps a search index in sync:

Core Concepts

The change event

Debezium's envelope carries the row before and after the change, the operation, and source metadata — enough for a consumer to do almost anything:

A tombstone is a separate message with a null value, published after a delete so log-compacted Kafka topics can drop the key entirely.

Log-based vs. query-based vs. trigger-based

Log-based wins on every axis that matters; the only reason to choose otherwise is a database or a managed service that won't grant you logical replication access.

How log-based CDC reads Postgres

Three moving parts to know:

wal_level = logical — required, and a restart. Managed providers expose it as a parameter (RDS: rds.logical_replication = 1).
Publication — which tables are streamed.
Replication slot — the server-side bookmark. It guarantees the WAL isn't recycled before the consumer reads it, which is exactly what makes an abandoned slot dangerous.

⚠️ Warning: A replication slot with no active consumer causes Postgres to retain WAL segments indefinitely, filling the disk and taking the database down. Always alert on pg_replication_slots.confirmed_flush_lsn lag, and drop slots for connectors you've decommissioned.

Snapshot then stream

A new connector has to reconcile history with the future. The standard sequence: take a consistent snapshot of existing rows (emitted as op: "r"), record the log position at the snapshot boundary, then stream everything after it.

For large tables, incremental snapshots (Debezium's signal-table mechanism) chunk the backfill and interleave it with live streaming, so you don't block replication for hours or hold long-running transactions on the source.

The Transactional Outbox

Raw table CDC couples consumers to your physical schema — rename a column and every downstream service breaks. The outbox pattern gives you CDC's atomicity guarantee while publishing versioned domain events you actually control.

Debezium's outbox event router unwraps these rows into clean messages on order topics. The outbox table can be pruned aggressively — the rows exist only to ride the WAL.

Why this beats a dual write

The dual write has no correct ordering. Publish-then-write can emit an event for a change that never commits; write-then-publish can commit a change nobody hears about. CDC removes the choice.

What CDC Is Used For

The migration use case is worth its own note: CDC is how you move a live database with minutes of risk instead of hours of downtime. Snapshot into the new store, stream changes until lag is near zero, freeze writes for seconds, verify, cut over.

Best Practices

Make every consumer idempotent

CDC delivery is at-least-once — a connector restart replays from the last committed offset. Dedupe on a stable key (the LSN, the transaction ID, or the row's own version column) and commit offsets only after the effect is durable. See Idempotency.

Key messages by primary key and let ordering do the work

Kafka guarantees order within a partition. Keying by the row's primary key puts every change to a given row in one partition, so pending → shipped → delivered can never be reordered. Random keys destroy this guarantee and produce inexplicable state.

Alert on replication lag, not just connector health

A connector can be "running" and hours behind. Monitor both sides:

Page when retained_wal crosses a threshold well below free disk. See PostgreSQL Monitoring and Alerting.

Prefer the outbox to raw table capture for service-to-service events

Raw CDC is right for replication and analytics, where you want the physical rows. For events other services consume, publish an explicit, versioned contract through the outbox so schema refactors stay internal.

Handle schema evolution deliberately

Use a schema registry with backward-compatible Avro/Protobuf, or JSON with an explicit version field. Adding a nullable column is safe; renaming or dropping one breaks consumers. Deploy consumers before producers on any change.

Never point CDC at the primary if you can avoid it

Reading from a replica keeps decoding work off the write path. Where the database requires slot creation on the primary (Postgres logical slots are not replicated to physical standbys by default), at minimum give the connector its own database user, its own connection limit, and a dedicated slot.

Exclude what you don't need

table.include.list and column blocklists keep throughput down and prevent PII from leaking into topics that aren't governed for it. Masking columns at the connector is far easier than scrubbing a Kafka topic.

Common Mistakes

Leaving an orphaned replication slot

Assuming exactly-once delivery

Committing offsets before the work is durable

Forgetting `REPLICA IDENTITY` and losing `before` images

Publishing raw internal tables as your public event contract

Treating CDC as a queue

CDC captures what changed in the database, not what the business intended. A status column flipping to cancelled doesn't tell you whether the customer cancelled or fraud review did. When intent matters, emit an explicit event through the outbox.

FAQ

Does CDC slow down my database?

Log-based CDC adds very little: the WAL is written regardless, and the connector is essentially another replication client. The real risks are operational — an unconsumed replication slot retaining WAL, or a heavy initial snapshot on a large table. Use incremental snapshots and monitor slot lag.

Do I need Kafka to use CDC?

No. Debezium Server can write to Kinesis, Pub/Sub, Pulsar, or Redis Streams, and Debezium Engine embeds directly in a JVM application. Managed options — AWS DMS, Google Datastream, Fivetran, Airbyte — skip the streaming layer entirely for warehouse loading. Kafka is common because durable, replayable, ordered, multi-consumer topics are exactly what change streams want.

CDC or the outbox pattern — which should I use?

They're complementary. CDC is the transport mechanism; the outbox is a schema pattern for what you send over it. Use raw table CDC for replication and analytics; use the outbox, delivered by CDC, for domain events other services consume.

How do I handle deletes in the warehouse?

Most warehouse targets soft-delete: apply the CDC delete as a _deleted = true flag plus a timestamp rather than removing the row, so historical queries and slowly-changing dimensions still work. Configure the sink's delete handling explicitly — the defaults vary and silently dropping deletes is a common data-quality bug.

Can I use CDC with a managed database?

Yes, with a setup step. RDS/Aurora Postgres need rds.logical_replication = 1; Cloud SQL has a logical decoding flag; Azure Database for PostgreSQL exposes wal_level. MySQL variants need row-based binlog. A few serverless offerings restrict replication slots — check before designing around it.

What about very large tables?

Use incremental snapshots. Debezium chunks the backfill via a signal table and interleaves chunks with live streaming, so you avoid a multi-hour blocking snapshot and can pause or resume it.