AWS Glue Iceberg Optimizer Alternatives for a Data Lakehouse

The AWS Glue Data Catalog's managed table optimizers are the path of least resistance for Iceberg maintenance on AWS. Three toggles, no Spark cluster to operate, $0.44 per DPU-hour. For a Glue-cataloged, Athena-queried, Parquet-only estate with batch write patterns, they are genuinely the right answer and this article is not trying to talk you out of them.

The reason you are reading this is that something broke that assumption. Compaction cannot keep pace with streaming ingestion and suspends itself. Trino and Athena both hit the same tables and the sort order helps neither. You have three catalogs and the optimizer only sees one. Or you finally looked at query planning time and realized nothing in the Glue optimizer rewrites manifests, which means nothing ever has.

This guide covers what the Glue optimizers actually do, with the real API parameters and documented defaults rather than the marketing summary; the specific constraints that cause teams to look elsewhere, including several that are genuine data-loss hazards rather than inconveniences; how to measure your real cost per terabyte instead of estimating it; and an honest survey of the alternatives, from Spark procedures through Flink, Trino, S3 Tables, the commercial platforms, four open-source projects, and dedicated control planes such as LakeOps that run maintenance as a coordinated loop across catalogs rather than as isolated jobs.

One framing worth setting up front, because it changes the conclusion: since the Glue Iceberg REST Catalog endpoint shipped, this stopped being an either/or decision. You can keep Glue's managed compaction doing bulk binpack work and attach an external tool over the REST protocol for the operations Glue does not offer. That hybrid is usually the lowest-risk answer, and it is the one most comparisons never consider.

The Short Version

What the Glue Optimizer Actually Does

There are three optimizer types, and the API enum has not grown: compaction, retention, and orphan_file_deletion. Two adjacent managed features are billed on the same meter and configured through the same catalog API, which is why they get mistaken for a fourth optimizer: column statistics generation, which writes Iceberg Puffin files, and materialized view auto-refresh. Neither is a TableOptimizer, and the statistics one matters enough that it gets its own treatment below.

Compaction

Merges small data files into larger ones using Iceberg's rewriteDataFiles. The documented trigger is specific, and it is worth quoting because most write-ups paraphrase it into something vaguer:

In the Data Catalog, the compaction process starts when a table or any of its partitions have more than 100 files. Each file must be smaller than 75% of the target file size.

Three things that matter in that sentence. It is evaluated per partition, not per table. The 100 is the default of a configurable parameter, not a constant. And the 75% is not an AWS invention, it is Iceberg's stock min-file-size-bytes default surfacing through the managed wrapper.

The real configuration surface is smaller than you might expect:

Note what is not there: there is no targetFileSizeMB API parameter. Target file size is controlled only through the Iceberg table property write.target-file-size-bytes, which defaults to 512 MB. This is a real difference from S3 Tables, which exposes target size directly in its maintenance API.

Note also that sort and z-order require you to have already defined a sort_order on the table. Glue will apply a sort order. It will never choose one, infer one, or update one.

Snapshot retention

Both criteria must be satisfied for a snapshot to survive: it must be within the retention period and within the minimum count.

Orphan file deletion

There is a safety semantic here that almost nobody documents and that surprises people badly: the optimizer only deletes files created after the optimizer itself was created. Files that predate the optimizer are never touched, regardless of age. If you enable orphan deletion on a table that has been accumulating cruft for two years, it will clean nothing until new orphans appear. Pre-existing orphans need a manual remove_orphan_files run.

The actual CLI shape

Configuration examples float around the internet with invented parameter names. The real structure nests the Iceberg config two levels deep:

aws glue create-table-optimizer \
  --catalog-id 123456789012 \
  --database-name analytics \
  --table-name events \
  --type compaction \
  --table-optimizer-configuration '{
    "roleArn": "arn:aws:iam::123456789012:role/GlueOptimizerRole",
    "enabled": true,
    "vpcConfiguration": { "glueConnectionName": "analytics-vpc-connection" },
    "compactionConfiguration": {
      "icebergConfiguration": {
        "strategy": "sort",
        "minInputFiles": 50
      }
    }
  }'

vpcConfiguration is the under-documented piece: it is how you run optimizers against tables reachable only inside a customer VPC.

Catalog-level defaults, and two traps

UpdateCatalog lets you set optimizer defaults for a whole catalog, which is the only thing that makes this manageable past a few dozen tables. Precedence runs table-level config, then catalog-level config, then Iceberg table property.

Two documented problems:

• Catalog-level inheritance silently does not work when Data Catalog metadata encryption is enabled. Every table must then be configured individually.
• AWS documents a known issue where tables without their own configuration may fail to inherit the disabled state from the catalog. Their own guidance is to audit the console and execution logs for optimizers running that you did not intend. That is a surprise-bill vector, not a cosmetic bug.

Pricing

$0.44 per DPU-hour, billed per second with a one-minute minimum per run. One DPU is 4 vCPU and 16 GB. AWS's worked example: 30 minutes on 2 DPUs costs $0.44.

The Glue 6.0 price reduction does not apply to optimizers. Glue 6.0 went GA in August 2026 with a headline 30% cut, and that cut is scoped to Spark ETL job DPU-hours on Glue 6.0. Table optimizers are not versioned, you cannot opt them into 6.0 pricing, and the optimizer meter still reads $0.44. Expect to see this conflated a lot over the next year.

The one-minute minimum deserves more attention than it usually gets. A high-churn table that triggers many small compaction runs pays a floor of 60 seconds of DPU each time, and the reports of Glue compacting fewer than ten files per run several times an hour are exactly this failure mode: you are billed a minimum-duration run to do almost nothing.

Which Iceberg version you get

Iceberg v3 deletion vectors and row lineage arrived in Glue 5.1, not 6.0. Glue 6.0 added the remainder of v3: VARIANT with shredding, geometry and geography types, and nanosecond timestamps. AWS Prescriptive Guidance explicitly lists Glue table maintenance as supporting v3, and states that after a v2 to v3 upgrade the next compaction removes the legacy v2 delete files.

The blocker nobody mentions: Athena cannot read Iceberg v3 at all. Glue 6.0 will happily write v3 tables that your Athena users cannot query. If Athena is in your read path, v3 is off the table regardless of what Glue supports, and since the v2 to v3 upgrade is atomic metadata-only but effectively irreversible once deletion vectors exist, this is a decision to make deliberately rather than discover.

Where it runs

Glue optimizers are available in 15 regions. Not available in us-west-1 (N. California) or any GovCloud region. If your lakehouse lives there, the evaluation is over before it starts.

Measure Your Real Cost Before You Decide

Almost every comparison of Glue optimizer costs against alternatives is built on estimates. It does not have to be, and this is the single most useful thing in this guide if you are trying to justify a change.

TableOptimizerRun returns an IcebergCompactionMetrics struct containing:

• dpu_hours — actual DPU-hours consumed by that run
• number_of_dpus — DPUs allocated, rounded up
• job_duration_in_hour
• number_of_bytes_compacted
• number_of_files_compacted

So your real cost per terabyte is not a guess:

$/TB = (Σ dpu_hours × $0.44) / (Σ number_of_bytes_compacted / 2^40)

Pull the history with ListTableOptimizerRuns and BatchGetTableOptimizer, aggregate across your top tables for the last 30 days, and you have a defensible number instead of a vendor's. Do this before you evaluate anything, because it also tells you which tables are burning DPU-hours to accomplish very little, which is usually a more actionable finding than the total.

Do not forget the S3 request bill

Compaction is read-many-small-files, write-few-large-files, plus listing. In us-east-1:

• PUT, COPY, POST and LIST: $0.005 per 1,000
• GET: $0.0004 per 1,000
• DELETE: free

Writes cost 12.5× more than reads per request, but reads dominate compaction because there are so many more of them, and they scale with input file count rather than bytes:

At ten million input files the S3 request bill rivals the compute bill. The modeling point that follows: compute cost scales with bytes, request cost scales with file count. Any comparison that models only DPU-hours understates the cost of pathologically fragmented tables, which are precisely the tables you are trying to fix.

LIST being billed at the PUT rate is also why orphan file deletion, a recursive prefix listing, is quietly the most expensive of the three optimizers on wide tables.

Where the Ceiling Is

I have grouped the documented considerations and limitations by how much they should worry you, rather than by feature area. The first group can destroy data.

Correctness hazards

Shared S3 locations cause cross-table deletion. If two Data Catalog tables point at overlapping S3 locations and both have retention or orphan optimizers enabled, one table's optimizer will delete files the other table still references. AWS documents this as "unrecoverable data loss from unintended deletion." Every table needs a unique, non-overlapping prefix, including across databases. This is the single most dangerous thing on the limitations page and it is almost never mentioned in comparison articles.

S3 lifecycle rules on Iceberg paths delete live files. A lifecycle expiration rule that touches an Iceberg prefix will remove manifests and data files still referenced by current snapshots. Exclude Iceberg paths from lifecycle rules entirely. AWS's suggested backstop is S3 versioning, which helps you recover but does not prevent the incident.

The one-million-file cap creates silent orphans. Retention and orphan deletion each delete at most 1,000,000 files per run. When snapshot expiry exceeds that cap, the overflow does not error, it becomes orphan files that a later orphan-deletion pass has to find. That is a two-stage cleanup dependency across two independently scheduled optimizers, and nothing sequences it.

Orphan deletion ignores everything older than the optimizer. Covered above, repeated here because it is the one that makes people think the optimizer is broken when it is behaving exactly as designed.

What the optimizer structurally does not do

Two of these deserve expansion.

Parquet-only compaction is buried on the compaction documentation page and absent from the limitations page, which is why nearly every comparison misses it. If any of your tables write Avro or ORC data files, Glue compaction simply does not apply to them. S3 Tables handles all three formats. This is a hard eligibility gate, not a performance consideration.

Puffin statistics are a subtlety, not a gap. It is tempting to write "Glue can't compute Iceberg statistics," and it would be wrong. Glue does generate Puffin NDV sketches, through a separate column-statistics feature configured via UpdateCatalog with ColumnStatistics.Enabled and ColumnStatistics.RoleArn. It runs weekly and samples 20% of records, and Athena needs use_iceberg_statistics=true on the table to consume them. So the accurate statement is narrower: the Glue table optimizer does not compute statistics, and the feature that does is on a separate schedule with no coupling to compaction. Since compaction is exactly what invalidates statistics, that decoupling is the actual problem. Your statistics can describe a file layout that a compaction run replaced four days ago.

No sequencing between the three

The three optimizers run independently. There is no dependency graph and no way to configure one.

Iceberg maintenance has a correct order, and it is not arbitrary. Expire snapshots first, so the later steps operate on a smaller file set and you are not rewriting files that are about to be dereferenced. Remove orphans next, including the ones expiry just created. Compact after that, against a clean current file set. Rewrite manifests last, so the metadata tree indexes the layout that now exists. Refresh statistics after that, so the optimizer's estimates match the files that will actually be read.

Glue gives you three of those five operations, on three independent schedules, in whatever order they happen to fire. The concrete waste is compaction rewriting files that snapshot expiry dereferences an hour later, and orphan cleanup running before expiry finishes and missing everything it was supposed to catch.

Architectural boundaries

• Single catalog. Only tables in the Glue Data Catalog. REST catalogs (Polaris, Nessie, Gravitino, Lakekeeper), Hive Metastore, and anything else get nothing. Production lakehouses increasingly span several, and each additional catalog means another maintenance story with its own blind spots.
• No cross-account tables and no cross-Region tables. Cross-account buckets work if the role has access, but cross-account tables and resource links do not.
• No S3 Express One Zone.
• Z-order does not support Decimal or TimestampWithoutZone.
• Observability is CloudWatch job telemetry. Success, failure, duration, DPU usage. It tells you a job ran. It does not tell you whether the table is healthy: file size distribution, manifest depth, delete-file accumulation, snapshot growth relative to write velocity, or whether the sort order still matches how the table is queried. Diagnosing that means querying Iceberg metadata tables in Athena per table and correlating manually, which is tractable at 20 tables and is not at 500.

The Concurrency Problem

This is the failure mode that pushes the most teams off the managed optimizer, so it is worth understanding precisely rather than as "compaction sometimes fails."

What actually happens

Reports from teams running hourly MERGE jobs alongside Glue auto-compaction describe roughly one in ten compaction runs failing with:

Compaction optimizer failed. Error: partial-progress.enabled is true but no rewrite
commit succeeded. Check the logs to determine why the individual commits failed. If
this is persistent it may help to increase partial-progress.max-commits which will
break the rewrite operation into smaller commits.

That message leaks two things AWS does not document anywhere else.

First, Glue's managed compaction runs rewrite_data_files with partial-progress.enabled = true. That is otherwise invisible internal behavior, and it explains a lot about the observed run patterns.

Second, and this is the part that makes it maddening, the remediation the error recommends is not available to you. partial-progress.max-commits is not exposed through the optimizer API, and the managed Spark job is not accessible. The error tells you to turn a dial you cannot reach.

Then it gives up

From the AWS documentation, verbatim:

When compaction operations fail four consecutive times, AWS Glue catalog table optimization automatically suspends the optimizer to prevent unnecessary compute resource consumption.

Re-enabling is manual, through the console or API. There is no auto-resume, no documented backoff curve, and no alarm. A suspended optimizer is silent, and the table degrades in the background until somebody notices the query latency. If you run Glue compaction on tables with concurrent writers, build a CloudWatch alarm on optimizer state right now, independently of anything else in this article.

Why the usual advice does not work

The standard internet recommendation for Iceberg commit conflicts is to drop from serializable to snapshot isolation. AWS states plainly, in its own guidance on concurrent write conflicts, that this does not help here:

For conflicts between streaming ingestion and compaction operations, which is one of the most common scenarios, snapshot isolation does not provide any additional benefits to the default serializable isolation.

The two conflict classes behave differently and only one is retryable:

The Iceberg-side properties you can actually set, with their defaults:

AWS's recommended baseline for tables under frequent concurrent writes raises retries to 10 with a 10-second max wait; for maintenance operations they suggest keeping 4 retries but raising the minimum wait to 1,000 ms. Set these as table properties and Glue's compaction will honor them, since it is running stock Iceberg underneath.

The structural fix

Retry tuning treats the symptom. The structural fix is to not attempt compaction on a partition that has an active writer, and to make the compaction window short enough that overlap is unlikely in the first place. Glue exposes neither: you cannot scope compaction with a WHERE predicate through the managed optimizer, and you cannot control when it runs. AWS's own documented mitigation is to serialize your writes against compaction, which negates most of the value of "automatic."

What a Replacement Actually Needs

Before looking at options, it helps to name the criteria the gaps above imply:

• Sequenced operations, where each step's output feeds the next, rather than three independent schedulers
• Manifest rewrite and statistics refresh as first-class operations, coupled to compaction rather than on separate cadences
• Query-aware layout, where sort keys come from the columns queries actually filter and join on, across every engine hitting the table
• Health-driven triggers based on table state, not fixed file counts, so a streaming table with 40,000 small files gets attention before a batch table slightly under target
• Writer-aware conflict handling, excluding hot partitions and backing off rather than failing four times and suspending
• Multi-catalog reach, because the catalog boundary is an accident of history, not a meaningful maintenance boundary
• Table health observability, not job telemetry
• An execution engine sized for the work, since compaction is I/O-bound read-merge-write and a general-purpose JVM cluster is structurally over-provisioned for it

The category this describes

What that list adds up to is not a better compaction job. It is a control loop, and nothing in the open lakehouse stack ships one.

A lakehouse control plane is the layer that runs it. It sits above the components you already have rather than replacing any of them: your catalogs, your Iceberg tables in S3, and your query engines all stay where they are. It attaches through standard catalog and Iceberg metadata APIs, so there is no data movement, no pipeline rewrite, and no change to the table format, which also means leaving is a matter of disconnecting it.

The loop has four steps. Sense table structure and query telemetry across every connected catalog and engine. Classify each table's health from those signals. Plan the operations and physical layout each table needs, in dependency order. Execute, then measure the result and feed it back into the next decision.

Glue's optimizers implement a fragment of the execute step. Everything else, the sensing, the classification, the sequencing, and the learning, is the part you are currently doing by hand or not at all. Hold that model while reading the options below, because it is the axis they actually differ on.

The Alternatives Landscape

Spark procedures: the DIY baseline

Calling rewrite_data_files, rewrite_manifests, expire_snapshots, remove_orphan_files, and compute_table_stats yourself gives you complete control and complete operational ownership. Every capability on the criteria list is achievable. You are building the scheduler, the health signals, the conflict handling, the sequencing, and the observability.

Worth knowing: Iceberg's own min-input-files default is 5, while Glue overrides it to 100. If your tables feel under-compacted on Glue relative to a Spark job you used to run, that difference is probably why.

This is where most teams land after outgrowing the managed optimizer, and it trades a maintenance problem for a platform-engineering problem. Whether that is a good trade depends entirely on whether you have the headcount to own it properly. The honest failure mode is not that it does not work, it is that it works until the person who built it changes teams.

Trino: the underrated option if you already run it

Trino's Iceberg connector exposes maintenance through ALTER TABLE ... EXECUTE, and it covers something Glue cannot do at all:

ALTER TABLE analytics.events EXECUTE optimize(file_size_threshold => '100MB')
    WHERE event_date < DATE '2026-09-01';

ALTER TABLE analytics.events EXECUTE optimize_manifests;

ALTER TABLE analytics.events EXECUTE expire_snapshots(retention_threshold => '7d');

ALTER TABLE analytics.events EXECUTE remove_orphan_files(retention_threshold => '7d');

Three points in Trino's favor. optimize accepts a WHERE clause, which is exactly the partition-scoped compaction AWS recommends and Glue does not expose. optimize_manifests clusters manifests by partitioning columns, closing the biggest Glue gap. And the retention floors (iceberg.expire-snapshots.min-retention and iceberg.remove-orphan-files.min-retention, both 7d) hard-fail rather than letting you delete live files, which is a real safety property Glue lacks.

The catch for AWS shops: Athena is Trino-derived but does not expose optimize_manifests. You need actual Trino, and you are then running maintenance on your query cluster, competing with user workloads.

Flink TableMaintenance: only if Flink is already yours

Flink can run maintenance inside the streaming job that writes the data, which eliminates a separate cluster and makes maintenance naturally writer-aware:

IcebergSink.forRowData(dataStream)
    .table(table)
    .tableLoader(tableLoader)
    .rewriteDataFiles(Map.of(RewriteDataFilesConfig.MAX_BYTES, "1073741824"))
    .expireSnapshots(Map.of(
        ExpireSnapshotsConfig.RETAIN_LAST, "5",
        ExpireSnapshotsConfig.MAX_SNAPSHOT_AGE_SECONDS, "604800"))
    .deleteOrphanFiles(Map.of(DeleteOrphanFilesConfig.MIN_AGE_SECONDS, "259200"))
    .append();

Available tasks are ExpireSnapshots, RewriteDataFiles, and DeleteOrphanFiles. Scheduling triggers include scheduleOnCommitCount, scheduleOnDataFileCount, scheduleOnDataFileSize, and several delete-file variants. None of them have defaults, so nothing is scheduled until you set one.

Two accuracy notes, because this area moves fast. ConvertEqualityDeletes, which resolves Flink's equality delete backlog into v3 deletion vectors through a staging branch, is real and genuinely important for CDC pipelines, but as of now it appears only in Iceberg's nightly documentation and not the stable release. And RewriteManifests is not merged for Flink; the pull request is open and committers have leaned toward waiting for native metadata compaction in Iceberg v4. So Flink maintenance does not close the manifest gap either.

Amazon S3 Tables

The most hands-off option, and the defaults are the opposite of Glue's: all maintenance is enabled by default, so you are being billed for compaction from the moment you create a table.

Strategy accepts auto, binpack, sort, or z-order, and auto picks sort when the table has a sort order defined and binpack otherwise. That is a genuinely better default than Glue's unconditional binpack. S3 Tables also compacts Avro and ORC, which Glue does not, and it exposes target file size directly.

The pricing model is where it gets interesting, and it changed substantially after AWS cut rates:

Object monitoring is the one to watch. It is a recurring per-object charge on everything in the bucket, which means a fragmented table is penalized twice: once at compaction time and then forever on monitoring. It penalizes exactly the condition compaction exists to fix. AWS's own example puts a 1 TB, 10,486-object table at $0.26/month monitoring against a $27.14 storage bill, which is small, but it scales with object count rather than bytes.

AWS's pricing text qualifies the $0.005/GB rate as applying to "default binpack compaction," which implies sort and z-order cost more. I could not find a published rate for those. Verify before modeling a sorted-table budget.

Other constraints: 10 table buckets per account per Region by default, 10,000 tables per bucket, namespaces cannot nest, Parquet row group size is fixed at 128 MB, and data type Fixed plus brotli and lz4 compression are unsupported. The only way to stop compaction charges is to explicitly disable compaction.

On throughput, published comparisons put S3 Tables managed compaction well behind alternatives on the same data, which matters less if you are optimizing for zero operational involvement and matters a lot if compaction needs to keep up with ingestion.

The Glue Iceberg REST catalog changes the question

This is the strategic point most comparisons miss entirely.

Glue exposes an Iceberg REST Catalog endpoint at https://glue.{region}.amazonaws.com/iceberg, configured with type=rest, warehouse=<account-id>, rest.sigv4-enabled=true, rest.signing-name=glue:

catalog.type=rest
catalog.uri=https://glue.us-east-1.amazonaws.com/iceberg
catalog.warehouse=123456789012
catalog.rest.sigv4-enabled=true
catalog.rest.signing-name=glue
catalog.rest.signing-region=us-east-1

That decouples engine from catalog. PyIceberg, Trino, Flink, DuckDB, Snowflake, or a purpose-built maintenance daemon can all reach Glue-cataloged tables over a standard protocol while Glue's managed optimizers keep running.

Which means "Glue optimizer versus an alternative" is often the wrong framing. A perfectly reasonable production setup keeps Glue compaction doing bulk binpack on the tables it handles well, and attaches an external tool over REST for manifest rewriting, statistics refresh, query-aware sort, and the tables Glue cannot touch. You are not migrating off Glue, you are filling in what it does not do.

Engine-integrated maintenance

Snowflake's arrangement has a trap worth flagging. For SNOWFLAKE_MANAGED external volume tables written only by Snowflake, compaction is bundled at no additional cost. Once an external engine performs DML or DDL through the Iceberg REST Catalog on or after 21 May 2026, that table begins incurring compaction credits permanently. One external write flips a free table into a billed one, and you can see it in ICEBERG_STORAGE_OPTIMIZATION_HISTORY.

The shared limitation across this whole category: each platform sees only its own workloads. In a multi-engine stack you get fragmented maintenance with no cross-engine coordination, which is the same problem as Glue with a nicer interface.

For completeness, since it still comes up: there is no standalone Tabular product. Tabular was acquired by Databricks in 2024 and its managed-maintenance thesis now lives inside Unity Catalog and Predictive Optimization.

Open source

This category is young and moving fast, so here is where each project actually stands rather than how its README reads:

• floe — Java, Apache 2.0. A policy-based orchestrator rather than an engine: it discovers tables, scores health into a "debt score" from snapshot count, small-file percentage, and delete ratio, then dispatches REWRITE_DATA_FILES, EXPIRE_SNAPSHOTS, REMOVE_ORPHAN_FILES, and REWRITE_MANIFESTS to Spark via Livy or to Trino. Multi-catalog across REST (Polaris, Lakekeeper, Gravitino), Hive, and Nessie, with a web UI. The most complete conceptual model of the four. Pre-1.0, and development has been quiet for several months.
• firn — Go, not Rust, despite how it is frequently described. A JVM-free daemon that shells out to DuckDB for compaction. Binpack, sort, z-order, snapshot expiry, orphan cleanup, across Lakekeeper, AWS Glue, Polaris, and Nessie, on S3, GCS, and Azure. Pre-1.0 and currently stale. Its README's claim of a "20-30x cost premium" for S3 Tables traces to a competitor's benchmark and predates the S3 Tables price cut; do not repeat it.
• bergman — Rust, single binary on iceberg-rust and DataFusion, no JVM. Notable for implementing manifest rewrite and dangling delete-file removal in its own commit layer, on the correct observation that nothing else cleans those up. Triggered rather than scheduled, with Prometheus metrics and a daemon mode. Actively developed but effectively a solo pre-release project; interesting design, not something to put in front of production data yet.
• nimtable/iceberg-compaction — Rust on DataFusion and iceberg-rust, handling positional and equality deletes. The strongest community signal of the four and the most actively maintained. Scope is narrower than it looks: full-table compaction only today, with partial and incremental compaction, binpack/sort/zorder strategies, and orphan deletion all still unchecked roadmap items.

None of these provide cross-engine query telemetry or lake-wide table health scoring. They are good options if you want to own the infrastructure and avoid another vendor, provided you are honest about maturity.

A structural note on where this is heading: Apache Polaris has become the de facto neutral catalog and explicitly does not execute maintenance, which is the gap these projects are racing to fill. Meanwhile Iceberg v4 discussions include native metadata compaction, which is part of why the Flink RewriteManifests work stalled. Manifest maintenance is a real gap today that the format itself may eventually close.

How a Control Plane Closes Each Gap

Taking the control-plane model from earlier and making it concrete against the specific Glue limitations above. LakeOps connects to the Glue Data Catalog through standard IAM credentials and runs alongside it, so Glue stays your catalog and your ETL engine.

Sequencing, as an actual dependency chain

The five-step order matters and Glue has no mechanism to express it. LakeOps runs snapshot expiration, then orphan cleanup, then compaction, then manifest optimization, with each step's output feeding the next. Cadence adapts per table to write velocity, so a streaming table may run the full cycle several times an hour while a weekly batch table runs once and an idle table is skipped entirely. The sequencing model is worth reading if you are building this yourself, because the ordering constraints are the part that is easy to get subtly wrong.

That also resolves the statistics decoupling described earlier. Puffin statistics are refreshed as part of the same chain, after the file layout changes, rather than on an independent weekly schedule that has no idea compaction happened.

Manifest rewrite as a first-class operation

The largest Glue gap is simply a supported operation here, with its own policy type rather than a Spark script you maintain separately. Because it is sequenced after compaction, the rewritten manifests index the file set that actually exists, which is the condition that makes manifest rewriting worth doing at all.

Query-aware sort instead of a sort order you guessed

Glue will apply a sort_order. It will never tell you what it should be. That question is unanswerable from inside any single engine, because the right layout depends on how Athena, Trino, Spark, and everything else query the table simultaneously.

LakeOps aggregates query telemetry across the connected engines, the AWS Glue integration covering Athena, Trino, Spark, Redshift, DuckDB, Snowflake, and Flink, and builds a per-table field-access profile from filter, join, and group-by frequency. Compaction then applies the sort order that maximizes min/max data skipping for that actual mix, and the strategy updates as access patterns move. When a new dashboard starts filtering on a different column, the next compaction pass reflects it rather than waiting for someone to notice.

Before a sort-order change touches production files, the candidate layout is validated on an Iceberg branch and compared against the baseline on real metadata, so a rewrite that would not improve scan reduction does not happen. That matters because sort-order changes rewrite every file in scope and are expensive to reverse.

Writer-aware conflict handling instead of suspension

This is the most direct answer to the auto-suspend problem. Rather than attempting compaction and failing four times, LakeOps monitors per-partition commit activity across connected engines and excludes partitions with active writers from the compaction scope, retrying them on a later cycle once commit frequency drops. When a commit does lose an optimistic-concurrency race, it backs off, re-evaluates partition state, and reschedules rather than retrying immediately into the same conflict.

There is also a mechanical contribution: a compaction pass that finishes in 221 seconds instead of 1,612 on the same 200 GB overlaps a far smaller window with your writers, which lowers conflict probability before any scheduling logic is involved. The commit conflict mechanics go deeper into why treating conflicts as expected behavior rather than as failures is the right operating model for multi-writer tables.

learn more:

Compaction Articles - LakeOps Blog

Iceberg compaction strategies, engines, and automation — binpack, sort, Z-order, streaming compaction, and production benchmarks.

lakeops.dev

Health-based triggers instead of a fixed file count

Glue's trigger is 100 files under 75% of target, per partition. That is a reasonable heuristic and it is the same heuristic for every table you own.

LakeOps evaluates each table across file count and size distribution, manifest depth, snapshot accumulation, delete-file ratio, partition skew, and sort-order alignment with observed query patterns, then prioritizes the most degraded tables first. The last signal has no equivalent in any threshold-based system, because it is not a property of the table at all. It is a property of the relationship between the table and its workload, and it degrades without a single byte being written.

An execution engine sized for the job

Compaction is I/O-bound read-merge-write. Running it on a general-purpose JVM cluster means JVM startup, GC pauses, executor provisioning, and idle capacity, compounded across hundreds of runs per day. LakeOps runs compaction on a Rust and DataFusion engine with bounded memory and disk spill instead. Published benchmarks on a 200 GB table:

Two production results from the same benchmark set: a 1,192 GB table that OOM'd Spark completed in 11 minutes, and a high-ingestion table went from 42,633 files to 69 in 138 seconds. Across 5.5 TB the file count dropped from 101,223 to 19,170. The full Spark replacement analysis has the methodology.

Set that against the $/TB number you calculated from dpu_hours earlier. That comparison is the whole business case, and you now have both sides of it measured rather than estimated. The broader cost picture also includes the storage side, since snapshot expiry and orphan cleanup reclaim bloat that never shows up in a compaction bill.

Multiple catalogs, one loop

Glue, REST catalogs (Polaris, Nessie, Gravitino, Lakekeeper), Hive Metastore, and S3 Tables are connected simultaneously, with every table discovered, health-scored, and maintained under consistent policies. If you run Glue for analytics, a REST catalog for streaming, and S3 Tables for a newer project, the catalog boundary stops being a maintenance boundary.

Policies instead of per-table CLI calls

Maintenance configuration cascades catalog-wide, then namespace, then table, with more specific policies overriding broader defaults. New tables inherit the right configuration from their namespace automatically. Compared with managing create-table-optimizer calls per table, and with the catalog-level inheritance caveats documented on the Glue side, this is the difference between configuration you can reason about and configuration scattered across an account.

Create a policy:

And let it run:

Table health, not job telemetry

Every maintenance operation is logged with before-and-after metrics: file counts, bytes reclaimed, duration, and the signal that triggered it. Severity-ranked insights flag excessive manifests, partition skew, and small-file buildup before query latency moves. The platform overview covers how observability, maintenance, compaction, governance, and routing compose into the single loop.

Side-by-Side

Making the Call

Stay with the Glue optimizers when

• Every table is in the Glue Data Catalog and every data file is Parquet
• Athena or Redshift is effectively your only query engine, so there is no cross-engine sort conflict to resolve
• Write patterns are batch, and compaction is keeping up without suspending
• You are under roughly 50 tables, where per-table configuration and manual metadata queries are still tractable
• Catalog-level defaults cover your policy needs, and metadata encryption is off so inheritance actually works

That is a real and common environment. Glue's optimizers are zero-infrastructure and cost nothing to leave running. Enable them, set catalog-level defaults, put a CloudWatch alarm on optimizer suspension, exclude your Iceberg prefixes from S3 lifecycle rules, and move on to a more interesting problem.

Supplement rather than replace when

• Glue compaction is working fine but planning time has grown and nothing rewrites manifests
• Your statistics are stale relative to compaction
• Most of your tables fit Glue's model and a handful do not

Connect over the Glue Iceberg REST catalog, leave the managed optimizers running, and add the operations Glue does not offer. This is the lowest-risk path and it is underused because most people frame the decision as a migration.

Replace when

• Streaming ingestion. Compaction that suspends itself is worse than no compaction, because it fails silently. If you are seeing repeated partial-progress.enabled is true but no rewrite commit succeeded, the remediation the error suggests is not reachable through the managed optimizer.
• Multi-engine reads. When Trino, Athena, Spark, and DuckDB hit the same tables, a static sort order set once at table creation is a guess that helps one workload and costs the others. The Athena cost analysis shows how directly bytes scanned tracks layout on scan-priced engines.
• Multiple catalogs. The catalog boundary is not a meaningful maintenance boundary, and maintaining a separate pipeline per catalog is how drift starts.
• Hundreds of tables. Per-table configuration and per-table metadata queries do not scale, and CloudWatch cannot tell you which table to look at first.
• Cost at scale. Run the dpu_hours calculation. If your measured $/TB is materially above what a purpose-built engine delivers, and you are compacting petabytes, that gap is a line item rather than a rounding error.
• Avro or ORC. Glue compaction does not apply. This one is binary.

The Bottom Line

The Glue optimizers are not a bad product. They are a narrowly scoped one: three independent operations, on one catalog, for one file format, triggered by one heuristic, with job-level telemetry. Inside that scope they work well and cost little.

What they are not is an operating model. Iceberg maintenance is a loop, not a set of jobs, and the parts Glue leaves out are the parts that make the loop work: sensing table health rather than job success, choosing a physical layout from how tables are actually queried, sequencing operations so each one's output feeds the next, and handling concurrent writers as normal rather than as failure. Those gaps are structural, not a roadmap item you can wait out.

The most useful next step is not choosing a tool. It is measuring what you have. Pull dpu_hours and number_of_bytes_compacted from ListTableOptimizerRuns for your top twenty tables, query the manifests and files metadata tables to see whether planning overhead and file distribution are actually where you assumed, and check whether any optimizer has quietly suspended itself. Most teams find at least one surprise, and the surprise usually decides the question.

If the answer is that you need the loop and not just the jobs, a control plane like LakeOps runs it on top of the Glue catalog you already have, without moving data or changing the table format. If the answer is that Glue covers you, that is a good outcome too, and now you know which limitations to watch for.