Lakehouse Tables: Snapshots, Compaction, and Retention

Lakehouse Tables and Compaction

Lakehouse table maintenance should preserve the table’s logical answers while changing how its files are organized and retained. Start by separating three jobs: making reads efficient, keeping needed history, and removing data that is no longer permitted or useful.

Apache Iceberg provides the concrete mechanisms here. Other table formats have their own transaction logs, maintenance APIs, and retention rules. A directory full of Parquet files is not enough to establish which files belong to a committed table version.

TL;DR

Use committed table metadata rather than listing a storage directory to identify rows.
Treat file compaction, manifest maintenance, snapshot expiration, and orphan cleanup as separate operations.
Preserve the oldest history required by active readers, recovery commitments, and approved policy.
Check logical results before and after a rewrite, including applicable delete semantics.
Verify physical deletion across retained snapshots and other copies when deletion is the goal.

Quick Example

This local Python example uses PyIceberg with its SQL/SQLite catalog dependency and PyArrow. It creates only a temporary teaching table, appends two batches, and compares current and historical scans. The checked environment used PyIceberg 0.12.0 and PyArrow 26.0.0.

Expected output is current=3 historical=2 snapshots=2. The first snapshot still exposes two rows after the second append. This tests actual local snapshot reads, not compaction, expiration, an object store, or a distributed writer. The temporary catalog and warehouse are cleaned up after the demonstration.

Core Concepts

Metadata defines visibility

A snapshot identifies a table state through metadata references. A file can exist physically without belonging to the current table, or remain necessary because an older retained snapshot references it. Directory age is therefore a poor deletion criterion. Iceberg’s maintenance guide distinguishes snapshot expiration from orphan removal. Maintenance

Evolution is not an eager rewrite

Iceberg tracks columns with field IDs and supports partition-spec evolution. New writes can use a new partition layout while older files keep their prior layout. Changing a spec does not immediately reorganize existing data. A team expecting an instant improvement must check which files its actual queries still scan. Evolution

Compaction changes physical work

Combining small files can reduce file-open and planning overhead. The useful target depends on query selectivity, partitions, ingestion cadence, and engine behavior. A larger file is not automatically a better file: an indiscriminate rewrite can increase bytes read for selective queries or consume more resources than it saves.

Choose the operation from the symptom

This diagnostic matrix is an editorial planning aid. Confirm the selected engine’s supported actions before running maintenance.

Begin with one bounded partition or representative table. Record a source snapshot, row-level or aggregate invariants, relevant delete files, and a comparison query set. Compare counts plus keys or value checks appropriate to the data; a count alone misses substituted rows. Keep the maintenance commit identity with the result.

Measure query planning and execution separately. A rewrite that reduces file count but leaves the bottleneck in a join may not justify its scheduling cost. Include ingestion and concurrent-reader behavior in a representative pilot before creating a recurring job.

Retention must account for every reader

Set the retention boundary from actual requirements rather than a copied interval. Inventory long-running queries, incremental consumers, table references, replication, legal holds, recovery procedures, and concurrent writers. Determine how each reader identifies its position and what happens if its required snapshot disappears.

Iceberg explicitly warns that overly aggressive orphan cleanup can remove files from in-progress writes and corrupt a table. It also warns about path comparison when storage authorities or path representations change. Prefer a supported inspection or dry-run capability when available, and review candidate paths against the actual deployment. Maintenance cautions

Compaction is not an erasure guarantee. Retained snapshots may still reference replaced files; storage versioning, backups, replicas, and exported datasets have separate lifecycles. A deletion workflow needs evidence for its declared scope, including exceptions and the time at which a retained copy will become inaccessible or physically disappear.

For a scheduled job, define who owns failures and how it avoids overlapping maintenance. Retain start/end snapshot IDs, candidate and affected file counts, job version, and validation outcomes. Stop on an unexpected logical difference or unresolved concurrent-write conflict rather than retrying destructive cleanup blindly.

Best Practices

Validate answers before celebrating smaller storage

Compare representative filters, aggregates, keys, and delete behavior. Keep correctness checks separate from performance observations.

Budget rewrite work

Limit scope and concurrency so maintenance does not overwhelm ingestion or interactive queries. Reconsider the schedule if little eligible data accumulates.

Make deletion evidence explicit

Name the table, retained history, backups, replicas, storage versions, policy exceptions, and responsible owners. Record unresolved copies instead of reporting an unsupported universal erasure.

Common Mistakes

Bad: Deleting files because their modification time looks old.

Correct: Use table-aware maintenance with validated references, writer timing, and supported retention rules.

Bad: Treating compaction as historical-data removal.

Correct: Verify snapshot expiration and each separate copy lifecycle when deletion is required.

Bad: Choosing a file size without representative reads.

Correct: Compare selective and broad queries, maintenance cost, and concurrency before adopting the target.

FAQ

Does compaction change query results?

A correct rewrite should preserve the intended logical result. Validate actual deletes, schema interpretation, and representative queries; matching file counts cannot establish correctness.

Does partition evolution rewrite existing files?

Iceberg’s metadata evolution does not eagerly rewrite older files. Plan a separate supported rewrite if changing existing layout is necessary.

Can expired snapshots be used for time travel?

Once the required snapshot/history is removed, that historical read is no longer an available recovery assumption. Verify the approved window before expiration.