Apache Iceberg Maintenance Tools: From Spark Procedures to Autonomous Control Planes

Every write to an Apache Iceberg table creates a new snapshot. Every snapshot pins references to data files. Over weeks of production workloads, this produces thousands of small files, bloated metadata, orphaned storage artifacts, and dangling delete records — all of which silently degrade query performance, inflate storage costs, and slow down every engine connected to the table.

The fix requires five distinct maintenance operations, executed in the right order, at the right frequency, across every table in the lake. The tooling landscape for this has matured substantially — from raw engine procedures to standalone Rust and Go binaries to fully autonomous control planes. This guide covers the complete spectrum, with enough technical depth to help you choose the right approach for your stack.

The five maintenance operations

Before evaluating tools, you need a clear picture of what maintenance actually involves. Iceberg maintenance is not a single operation — it's a coordinated pipeline of five distinct actions, each solving a different structural problem.

Here's an example of such operations using an autonomous and intelligent Lakehouse control plane:

Managed Iceberg in 2026: Autonomous Data Lake - LakeOps Blog

A deep dive into nine components of a modern Iceberg optimization platform — full-stack observability, query-aware compaction, snapshot lifecycle management, manifest optimization, orphan file cleanup, organization-wide policies, multi-engine query routing, agentic AI enablement, and layout simulations — with product walkthroughs showing how LakeOps delivers autonomous table maintenance, up to 28x faster compaction, and up to 80% cost reduction.

lakeops.dev

1. Snapshot expiration

Every commit — write, update, delete, compaction — creates a new snapshot. Snapshots enable time travel and rollback, but they also prevent data files from being garbage-collected. A streaming table with 5-minute commits accumulates over 8,600 snapshots per month.

expire_snapshots removes snapshots older than a retention threshold and deletes the manifest files and manifest lists that were exclusively referenced by those expired snapshots. Data files that are no longer referenced by any retained snapshot become eligible for orphan cleanup (they are not automatically deleted by expiration alone on all engines — always pair with remove_orphan_files).

Key parameters:

• older_than — the retention boundary. Set this to at least 2× the duration of your longest-running query. A snapshot expired mid-scan produces a FileNotFoundException that surfaces minutes later, making root cause analysis painful.
• retain_last — the minimum snapshot count regardless of age. Never set this to 1 — that leaves zero rollback targets. For streaming tables, 25–100 is typical.
• stream_results — set to true on large tables. Without it, the full list of deleted files is collected in the Spark driver's memory before being returned, which causes driver OOM on tables with millions of expired files.

2. Orphan file cleanup

Distributed engines produce orphan files — data and metadata files that exist in object storage but are not referenced by any snapshot. They accumulate from failed Spark tasks, aborted compaction jobs, and commit conflicts. On active tables, orphan storage can grow to hundreds of gigabytes without anyone noticing.

remove_orphan_files lists every file in the table's storage location and compares what exists against what the metadata tree references. Anything unreferenced and older than the retention interval is deleted.

The default retention is 3 days, but production teams use 7+ days. The reason: a Spark job that writes files at the start of a 4-hour run but commits at the end has unreferenced files for 4 hours. A retention shorter than that window deletes live data mid-write, silently corrupting the table.

Three less-obvious failure modes to watch for:

• URI scheme mismatch — if your metadata records paths as s3:// but the listing returns s3a://, the diff considers every file an orphan and deletes your entire table's data. Always verify schemes match before first run, and use prefix_mismatch_mode => 'ERROR' (the default since recent Iceberg versions) to catch this.
• S3 rate limiting — orphan cleanup issuing thousands of concurrent deletes can trigger S3's 3,500 DELETE/s per-prefix throttle, producing SlowDown errors that affect production reads. Use max_concurrent_deletes to cap throughput.
• Driver OOM on large tables — set stream_results => true to stream file lists in partitions rather than collecting them all in the driver.

3. Data file compaction

Small files are the most visible maintenance problem. Each file adds metadata overhead, and each file open during a scan adds latency. A partition with 500 files averaging 4 MB each is orders of magnitude slower to scan than the same data in 8 files of 256 MB.

rewrite_data_files reads small files and rewrites them into larger, optimally-sized targets. Three strategies are available:

• Binpack — merges files without changing row order. Fast, low-risk, and sufficient when queries scan full partitions.
• Sort — rewrites files with rows physically ordered by specified columns. Maximizes data skipping via Parquet min/max statistics. More expensive (requires a full shuffle) but delivers the largest performance gains for filtered queries.
• Z-order — interleaves sort columns for multi-dimensional range queries. Useful when queries filter on multiple independent columns.

Target file size depends on workload: 64–128 MB for point-lookup tables, 256–512 MB for analytical scans. Most production tables land at 256 MB as a default.

Two parameters are critical for avoiding OOM during compaction: max-file-group-size-bytes controls how much data is processed in a single rewrite group — set it too high on a partition with thousands of small files and your Spark executors run out of memory. rewrite-job-order (e.g., bytes-asc) lets you process smallest groups first, catching configuration problems early before committing resources to large rewrites. Note that Spark disables Adaptive Query Execution (AQE) during rewrite_data_files, so size your shuffle resources explicitly rather than relying on auto-tuning.

4. Manifest rewriting

Manifests are the metadata index — they map which data files exist in which partitions with which column statistics. When writes arrive in a pattern that doesn't match query patterns (e.g., data written hourly but queried by customer ID), manifests become fragmented. Query planners must scan more manifests than necessary, adding seconds of planning overhead on large tables.

rewrite_manifests reorganizes manifest files to align with the actual file layout, reducing query planning time. The operation is metadata-only — it rewrites Avro manifest files but never touches data files, making it fast (seconds on most tables) and low-risk. The conflict window with concurrent writers is correspondingly narrow, so run it immediately after compaction before the next batch of writes lands.

5. Delete file maintenance

Iceberg's merge-on-read mode writes position delete files instead of immediately rewriting data files on row-level deletes. This is efficient for small deletes but accumulates overhead: every scan must reconcile delete files against base data at read time.

rewrite_position_delete_files compacts small delete files and filters out dangling delete records — entries pointing to data files that no longer exist after compaction. This is the most frequently overlooked maintenance operation, and on tables with frequent row-level deletes, it can account for a significant portion of read overhead.

Why sequencing matters

These operations have dependencies. Running them in the wrong order wastes compute or misses reclaimable storage:

1. Expire snapshots first — dereferences files held exclusively by old snapshots, converting them from "referenced" to "orphan-eligible." Without this step, orphan cleanup misses the largest category of reclaimable storage entirely.
2. Remove orphans second — catches everything expiration just released, plus any debris from failed writes. Reduces the file set before compaction.
3. Compact data files third — operates on a clean, minimal dataset. Compacting files that expiration would have removed is the single largest source of wasted compute in maintenance pipelines.
4. Rewrite manifests last — reflects the final compacted layout. Rewriting manifests before compaction produces a metadata alignment that becomes stale the moment compaction runs.

Running these as independent cron jobs on separate schedules is the most common source of maintenance collisions and wasted compute. See Automating Apache Iceberg Table Maintenance for a detailed treatment of the sequencing rationale.

Commit conflicts during maintenance

Iceberg uses optimistic concurrency control (OCC) for all metadata commits. When a maintenance operation and a writer commit simultaneously, one loses the race and must retry. On streaming tables producing multiple commits per minute, compaction jobs that run for 10+ minutes will almost certainly hit at least one conflict.

The critical distinction: catalog commit conflicts (two writers race to swap the metadata pointer) are transient and resolve through automatic retries. Data conflicts (compaction rewrites files in a partition that received new writes during the rewrite) cause ValidationException under serializable isolation and require either custom retry logic or switching to snapshot isolation.

Production guidance:

• Exclude hot partitions from compaction. On time-partitioned tables, compact WHERE event_date < current_date() — never today's partition while a streaming writer is appending to it. This single filter eliminates the most common source of compaction failures.
• Use snapshot isolation for tables with concurrent streaming writes and compaction. It allows appends to the same partition that compaction is rewriting, as long as the appended files don't overlap with the rewritten set.
• Increase commit.retry.num-retries on streaming tables from the default 4 to 10–20. The retry only repeats the metadata commit, not the entire rewrite.
• Watch for retry storms. Contention causes retries, retries increase catalog load, higher catalog load slows commits, which widens the contention window. If your catalog request rate climbs while successful commit rate falls, reduce concurrency rather than increasing retries.

Engine-native maintenance: Spark

For most teams, Spark is the first maintenance tool they use with Iceberg. The Iceberg connector exposes stored procedures covering every core operation:

-- 1. Expire snapshots older than 7 days, keep at least 10
CALL catalog.system.expire_snapshots(
  table => 'db.events',
  older_than => TIMESTAMP '2026-09-09 00:00:00',
  retain_last => 10,
  stream_results => true  -- prevents driver OOM on large tables
);

-- 2. Remove orphan files older than 7 days (always dry-run first)
CALL catalog.system.remove_orphan_files(
  table => 'db.events',
  older_than => TIMESTAMP '2026-09-09 00:00:00',
  dry_run => true,
  stream_results => true,
  max_concurrent_deletes => 1000  -- throttle to avoid S3 rate limits
);

-- 3. Compact data files (exclude today's hot partition)
CALL catalog.system.rewrite_data_files(
  table => 'db.events',
  strategy => 'binpack',
  where => 'event_date < current_date()',
  options => map(
    'target-file-size-bytes', '268435456',
    'min-input-files', '5',
    'max-file-group-size-bytes', '3221225472',
    'partial-progress.enabled', 'true',
    'partial-progress.max-commits', '10'
  )
);

-- 4. Rewrite manifests
CALL catalog.system.rewrite_manifests(
  table => 'db.events'
);

-- 5. Clean up dangling position delete files
CALL catalog.system.rewrite_position_delete_files(
  table => 'db.events',
  options => map('rewrite-all', 'true')
);

What Spark does well: Full control over every parameter. Support for all five operations. The partial-progress.enabled option is critical for production — it commits compaction incrementally, so a failure at 80% doesn't lose all prior work. The stream_results option prevents driver OOM on tables with millions of files.

What Spark doesn't do: Spark provides procedures, not a maintenance system. There is no scheduling, no health detection, no multi-table orchestration, no sequencing logic, and no awareness of what other engines or operations are running against the same table. You build all of that yourself — typically as Airflow DAGs, cron jobs, or custom scripts. That works at 10 tables. At 500, it becomes its own infrastructure project.

The other major limitation is cost: Spark requires a running cluster. Running maintenance on a dedicated Spark cluster that sits idle between jobs, or sharing a cluster with production queries and risking commit conflicts, is a tradeoff every team manages differently.

Engine-native maintenance: Trino and Flink

Trino

Trino exposes maintenance through ALTER TABLE ... EXECUTE, keeping operations inside the same catalog, permissions model, and deployment pipeline as queries:

ALTER TABLE db.events EXECUTE optimize
  WHERE event_date < current_date - INTERVAL '1' DAY;

ALTER TABLE db.events EXECUTE expire_snapshots(retention_threshold => '7d');

ALTER TABLE db.events EXECUTE remove_orphan_files(retention_threshold => '7d');

The advantage is simplicity — no separate Spark cluster for maintenance. The disadvantage is that maintenance competes with query resources on the same Trino cluster, and Trino's maintenance surface is narrower than Spark's (no sort compaction, no position delete file rewriting, fewer tuning knobs).

Flink

Flink provides native streaming maintenance through the TableMaintenance API. Maintenance runs as a Flink streaming job — triggered by commit counts or file thresholds rather than external schedules. Since Iceberg 1.12, external lock factories are deprecated; Flink uses its own coordination locks by default:

// Standalone maintenance job (Flink-managed coordination lock)
TableMaintenance.forTable(env, tableLoader)
    .uidSuffix("my-maintenance-job")
    .rateLimit(Duration.ofMinutes(10))
    .lockCheckDelay(Duration.ofSeconds(10))
    .add(ExpireSnapshots.builder()
        .scheduleOnCommitCount(10)
        .maxSnapshotAge(Duration.ofDays(7))
        .retainLast(10)
        .parallelism(8))
    .add(RewriteDataFiles.builder()
        .scheduleOnDataFileCount(10)
        .targetFileSizeBytes(128 * 1024 * 1024)
        .partialProgressEnabled(true)
        .partialProgressMaxCommits(10))
    .add(DeleteOrphanFiles.builder()
        .minAge(Duration.ofDays(5)))
    .append();

Flink also supports post-commit maintenance directly in IcebergSink — compaction, expiration, and orphan cleanup run automatically after each data commit, eliminating external scheduling entirely for Flink-written tables:

IcebergSink.forRowData(dataStream)
    .table(table)
    .tableLoader(tableLoader)
    .rewriteDataFiles(Map.of(
        RewriteDataFilesConfig.MAX_BYTES, "1073741824"))
    .expireSnapshots(Map.of(
        ExpireSnapshotsConfig.RETAIN_LAST, "5",
        ExpireSnapshotsConfig.MAX_SNAPSHOT_AGE_SECONDS, "604800"))
    .deleteOrphanFiles(Map.of(
        DeleteOrphanFilesConfig.MIN_AGE_SECONDS, "259200"))
    .append();

For tables with row-level deletes in upsert/CDC workloads, Flink also provides ConvertEqualityDeletes — a maintenance task that rewrites expensive equality deletes into position deletion vectors (requires Iceberg format v3). This can significantly reduce read amplification on tables with frequent upserts.

When engine-native works

Engine-native tools are a good fit when:

• You have a small number of tables (under ~50)
• A single engine handles both writes and maintenance
• Your team has the capacity to build and maintain orchestration (DAGs, cron, monitoring)
• You don't need cross-engine coordination

They start to break down when multiple engines touch the same tables, when table counts grow beyond what manual orchestration can cover, or when the operational cost of maintaining the maintenance system itself becomes significant.

Standalone maintenance tools

A newer category of tooling has emerged: standalone maintenance binaries that run outside any query engine. They connect directly to Iceberg catalogs and object storage, execute maintenance operations, and exit. No Spark cluster, no Trino session, no JVM in some cases.

Bergman

Bergman is a Rust-native maintenance engine built on iceberg-rust and Apache DataFusion. One static binary. No JVM.

It supports all core operations — compaction, snapshot expiration, orphan cleanup, manifest rewriting, and dangling delete removal — driven by declarative TOML policies:

bergman inspect   # table health: file counts, sizes, delete ratios
bergman plan      # what maintenance would do, and why
bergman run       # execute the plan (same code path as plan)

Bergman's design is library-first: the binary is a thin wrapper around a public Rust API, so catalogs and control planes can embed the engine in-process. Destructive operations (orphan deletion, compaction, file cleanup) are off by default and must be explicitly opted into via policy. It currently supports REST catalogs and is early-stage — all operations are functional but it has not yet seen large-scale production use.

Firn

Firn is a single Go binary that provides writer-agnostic maintenance — compaction (binpack, sort, and z-order via a DuckDB subprocess), snapshot expiration, and orphan cleanup. It supports AWS Glue, Lakekeeper, Apache Polaris, and Nessie catalogs, with S3-compatible, GCS, and Azure storage backends. Firn positions itself as an open-source alternative to AWS S3 Tables' automatic maintenance — the same operations, without AWS lock-in or the cost premium. Pre-v1.0.

iceberg-zamboni

iceberg-zamboni uses PyIceberg for metadata and DuckDB for sorting, providing all six maintenance operations as a single Python process. It supports Z-order clustering, partition evolution, dangling-delete removal, and manifest rewriting — locally by default, with optional Trino or Spark Connect backends for scale. Install with pipx install iceberg-zamboni. Declarative table configs describe the target layout; every mutating command requires --yes to execute.

Ice-keeper

Ice-keeper wraps Iceberg's Spark procedures and orchestrates them across many tables using rules stored in Iceberg tblproperties. It handles table discovery, parallelizes work across large inventories, and records every action in a journal for auditability. It runs wherever Spark runs — the benefit is multi-table orchestration, not engine independence.

Floe

Floe is a policy-based maintenance system that adds declarative policies over Spark or Trino. Define rules matching tables by glob or regex pattern, set cron schedules or manual triggers, and Floe orchestrates compaction, snapshot expiration, orphan cleanup, and manifest optimization across multiple catalogs (REST, Polaris, Lakekeeper, Gravitino, DataHub, Nessie, Hive). Policies support priority-based resolution when multiple patterns match the same table and optional health-based trigger conditions. Apache 2.0 licensed.

What standalone tools add

The common thread: these tools treat maintenance as a first-class concern separate from query execution. They add table discovery, policy-driven configuration, multi-table orchestration, and operational logging — the layer that raw engine procedures leave entirely to you.

What they generally don't provide: health-driven triggering (knowing when a table needs maintenance based on its actual state), cross-engine awareness, intelligent sort-order selection based on query patterns, or the ability to adapt cadence to write velocity.

Platform maintenance layers

If you run within a managed platform, maintenance tooling is increasingly built in:

Platform tools excel when you're fully committed to one ecosystem. They break down when your data lake spans multiple engines, catalogs, or clouds — which is increasingly common in production architectures that use Iceberg specifically to avoid vendor lock-in.

Lakehouse control planes

The gap in all the approaches above is the same: none of them treat maintenance as a lake-wide coordination problem. Engine procedures don't know about other engines. Standalone tools orchestrate operations but don't understand query patterns. Platform layers are confined to their ecosystem.

A lakehouse control plane
bridges your catalogs, engines, and storage into a single operational layer — the coordination fabric that individual components were never designed to provide. LakeOps is built specifically for this role.

Learn more:

What Is a Data Lakehouse Control Plane? - LakeOps Blog

What is a data lakehouse control plane? An intelligent layer that autonomously maintains, optimizes, and governs Apache Iceberg tables across all engines.

lakeops.dev

Health-driven vs. schedule-driven

The fundamental architectural difference between a control plane and a cron-based approach is what triggers maintenance.

Scheduled maintenance runs on fixed intervals regardless of table state. A table that receives no writes still gets compacted on Tuesday. A streaming table that accumulated 10,000 small files since the last run waits until the next scheduled window.

Health-driven maintenance evaluates each table's structural signals continuously — file count per partition, average file size relative to target, snapshot count, manifest depth, delete-file ratio — and triggers operations only when indicators cross thresholds. Streaming tables may be maintained hourly. Batch tables daily. Idle tables are skipped entirely. The cadence adapts to write velocity, not wall-clock time.

How LakeOps approaches the problem

LakeOps plugs into your existing Iceberg catalogs (REST, Glue, Polaris, Nessie, S3 Tables) and engines (Spark, Trino, Flink, Snowflake, Athena, DuckDB) through their standard APIs. No infrastructure changes, no code changes — only metadata is processed, and your data stays in your storage account.

For maintenance specifically, the system:

• Continuously assesses every table's structural condition — file layout, snapshot accumulation, manifest depth, write velocity — and flags degradation before it reaches queries. The table health scoring model evaluates signals that individual engine procedures have no visibility into, surfacing the tables that need maintenance most urgently.

• Triggers Operations based on tresholds or events and sequences operations in dependency order per table: snapshot expiration → orphan cleanup → compaction → manifest rewriting. Each step runs to completion before the next begins, eliminating the conflict scenarios and wasted-compute problems described above. There is no possibility of expiration conflicting with in-flight compaction, because they never run concurrently on the same table.

• Applies query-aware sort orders during compaction. Rather than blindly merging small files, LakeOps watches which columns your queries actually filter, join, and group on across all connected engines — then physically re-sorts data to match. Engines skip entire file groups via min/max pruning instead of scanning everything. The sort strategy adapts as query patterns evolve, and proposed layout changes are validated on Iceberg branches before being applied.

• Runs compaction on a purpose-built engine (Rust + Apache DataFusion) rather than requiring a Spark cluster. This eliminates JVM startup latency, executor provisioning, and resource contention with production queries — a significant operational and cost reduction.

• Governs maintenance through policies rather than per-table configuration. Thresholds, retention windows, and sort strategies propagate through a namespace hierarchy, so new tables comply automatically. Execution modes range from fully autonomous to manual approval.

The platform page shows the full maintenance pipeline in detail — from health scoring through coordinated execution to the event trail that logs every operation's duration, file impact, and bytes reclaimed.

Real example from a 7k tables Iceberg Lakehouse:

For teams dealing specifically with small file accumulation or the operational burden of table cleanup at scale, the control plane approach replaces per-table cron jobs with a single system that handles the entire maintenance lifecycle across the lake.

Choosing the right approach

The right tool depends on three variables: how many tables you manage, how many engines touch them, and how much operational overhead your team can absorb.

There's no single correct answer. The common mistake is underestimating how quickly the operational surface area grows. Ten tables with four operations each, sequenced correctly, monitored for failures, adapted to write patterns, across two engines — that's already a non-trivial system to build and operate.

Practical implementation checklist

Regardless of which tool you choose, these principles apply:

• Always dry-run orphan cleanup before executing. A misconfigured older_than or a URI scheme mismatch can delete live data. Verify the count is reasonable — if it's close to your total file count, something is wrong.
• Expire before compact. This is the single most impactful sequencing rule. Compacting files that expiration would have removed wastes the most compute.
• Enable partial-progress for compaction on any table larger than a few gigabytes. Without it, a single OCC conflict at 90% completion loses all prior work.
• Exclude hot partitions from compaction. Use where => 'event_date < current_date()' or equivalent. Compacting partitions with active streaming writers is the primary cause of commit conflicts and wasted compute.
• Set retain_last to at least 2. Setting it to 1 leaves zero rollback targets after the current snapshot. For streaming tables, 25–100 is appropriate.
• Use stream_results => true for expire_snapshots and remove_orphan_files on any table with more than a few thousand files. Without it, the full file list is collected in the Spark driver's memory.
• Use a 7+ day orphan retention. The 3-day default is too aggressive for environments with long-running Spark jobs or Flink recovery scenarios. Verify URI schemes match before first run.
• Monitor compaction output, not just completion. A compaction job that succeeds but produces files 10× larger than your target is masking a configuration problem.
• Run rewrite_position_delete_files regularly on tables with row-level deletes. Dangling deletes are invisible in most monitoring but accumulate meaningful read overhead.

Where the ecosystem is heading

The Iceberg maintenance ecosystem is converging on a clear pattern: from manual procedures toward automated, health-driven systems. Flink's native TableMaintenance API builds maintenance into the streaming pipeline. Standalone tools like Bergman, Firn, and iceberg-zamboni remove the JVM dependency entirely. Control planes like LakeOps close the loop between table health signals and maintenance execution across the full lake.

The underlying insight across all of these is the same: maintenance is a continuous structural concern, not an ad-hoc operational task. The teams that treat it as infrastructure — with proper sequencing, conflict awareness, adaptive cadence, and per-table tuning — avoid the silent degradation that catches everyone else by surprise six months into production.