Your Iceberg metadata directory is growing faster than your data

Your Iceberg .metadata/ directory is growing faster than your data.

Every commit writes a new snapshot. A streaming ingest writing every 30 seconds is 2,880 snapshots a day. After a month that’s around 86,000 files in one directory, each referencing manifests, referencing data files.

Nothing in Iceberg cleans this up automatically. The default behavior is to keep every snapshot, forever. The .metadata/ directory eats storage, slows query planning, and inflates your storage listing bill long before you ever notice.

The fix is three operations you should be running on a schedule:

• expire_snapshots removes old snapshots by time or count. Keep a week of history, not all of history
• rewrite_manifests compacts the small manifest files that pile up. Fewer files, faster planning
• remove_orphan_files deletes files that no live snapshot references, the ones that survive failed commits

You wouldn’t run a Delta table without OPTIMIZE and VACUUM. Iceberg needs the same maintenance, just under different names.

Most teams find out the hard way. Either the .metadata/ directory grows larger than the actual table, or partition pruning suddenly takes seconds instead of milliseconds.

What does your Iceberg maintenance schedule look like?