Original Japanese article: Icebergは本当にベンダーロックインを解消したのか考えてみた

Introduction

I'm Aki, an AWS Community Builder (@jitepengin).

With the spread of Apache Iceberg, I hear more and more that data platforms have become more open.
"Iceberg is an open format, so you can avoid vendor lock-in" is an explanation I see often, and one I've given myself.

Working with data platforms built on AWS, as well as Databricks, Snowflake, and Microsoft Fabric, I've found plenty of situations where that explanation holds true.
But I started wondering how far it really goes, so this time I want to sort out which kinds of lock-in Iceberg has solved, and which it hasn't.

To give the conclusion up front: what Iceberg achieved is data portability.
Governance, meaning permissions, auditing, lineage, and sharing, does not travel with the data in the same way.
My current view is that lock-in hasn't disappeared; it has simply moved up to a higher layer.

By the way, I'm a big fan of Unity Catalog!

Where Did Lock-in Used to Live?

Let's start by looking back at how things used to be.

In traditional data warehouses and data lakes, the file format, the way metadata was managed, transaction management, partition management, and the access APIs were all closed off inside the service.
In that structure, "the data living inside that service" was itself the lock-in.
Moving to a different environment meant starting by extracting all of the data and converting it.

Open table formats are what solved this.
Data files sit in cloud storage such as S3 as Parquet, and metadata sits right alongside them in an open specification.
Because both specifications are public, neither is locked into a vendor-specific format.

So what, then, is not open?

The Catalog Defined by the Iceberg Spec Is Actually Very Narrow

Let's go back to the spec and check what role the catalog plays in Iceberg.

The "Overview" section under "Catalog" in the official Apache Iceberg documentation explains that creating, dropping, and renaming tables are the catalog's responsibilities, and that its most important responsibility is tracking a table's current metadata and providing it when the table is loaded.
It also says that multiple kinds of compute engines can share a common data layer by sharing the same catalog.

https://iceberg.apache.org/terms/

So how does that "current metadata" get updated?
That's covered in the Table Spec.

https://iceberg.apache.org/spec/

Every change to table state creates a new metadata file, which replaces the old metadata through an atomic swap.
Writers optimistically create a metadata file on the assumption that the current version won't change before they commit, and then commit by swapping the table's metadata pointer from the base version to the new version.
This is what's known as compare-and-swap (CAS): even when multiple writers commit at the same time, updates based on stale state can be detected and treated as conflicts.

Here's what that looks like as a diagram.

   Catalog
┌──────────────────────────────────────┐
│  table_a → s3://.../v3.metadata.json │
└──────────────────────────────────────┘
              │
              │ Pointer to current metadata
              ▼
┌──────────────────────────┐
│  metadata.json           │  ← Open
│    schemas / snapshots   │
│    partition-specs       │
│  manifest list / files   │  ← Open
└──────────────────────────┘
              │
              ▼
┌──────────────────────────┐
│  Parquet / S3            │  ← Open
└──────────────────────────┘

Schemas, snapshots, and partition specs all actually live inside metadata.json.
Managing the list of snapshots used for time travel is not the catalog's responsibility either.

One thing worth pointing out here: the same Table Spec states that the atomic operation used to commit metadata depends on how tables are tracked, and that the spec does not standardize it.
In other words, what the Table Spec defines stops at "swap the pointer atomically"; where and how that pointer is kept is left to the catalog implementation.
That is the minimum set of catalog responsibilities visible from the table spec.

The REST Catalog Spec Broadened the Catalog's Role

At the implementation level, though, things don't stop there.

To support more languages and engines, the Iceberg project created a common API called the REST Catalog Protocol.
And with the REST Catalog, the commit mechanism itself changes.

https://iceberg.apache.org/rest-catalog-spec/

According to the "Commit protocol" section of the REST Catalog Protocol page, unlike client-side catalogs, a REST client neither writes metadata files nor swaps the pointer itself.

https://iceberg.apache.org/docs/nightly/rest-protocol/

Instead, the client sends the server the conditions that must still hold for the commit to be valid (Requirements), along with the metadata changes themselves (Updates).
The server checks every requirement against the current metadata and, if they all hold, applies the updates and writes the new metadata.

In other words, with the REST Catalog, responsibility for committing has moved from the client to the catalog.
The same page explains that because the server owns the commit logic, it enables server-side conflict resolution, multi-table commits, and secure table sharing through credential vending or remote signing.

Let's also look at credential vending.
The server can include short-lived, table-scoped storage credentials (storage-credentials) in its load-table response.
However, whether to vend credentials is up to the server, and a separate mechanism called remote signing, where the server only signs requests, is also defined.
So the accurate reading seems to be that the spec defines access delegation mechanisms as optional.

What the spec does not define, on the other hand, is the authorization model behind that delegation (how "who can access what" is expressed).
Even though how you call credential vending is standardized, the logic that decides "who gets which permissions," and the way that is expressed, are left to each catalog implementation.

This looks like the boundary with what real-world data catalogs actually cover.

What Real-World Data Catalogs Cover

When you look at catalogs as actual products, they take on even more.

They offer a wide range of capabilities: namespace management, serving table metadata, credential vending, access control including row- and column-level security, tag-based and attribute-based permissions, data discovery, lineage, audit logs, data sharing, and integration with external tables and other catalogs.

This is not at all a criticism of the products.
What we actually expect from a catalog in production is, if anything, these additional capabilities.
A pointer plus a credential vending endpoint doesn't settle the authorization logic of who can read which table, and it doesn't give you auditing either.

The problem is that because these additional capabilities aren't open, each implementation ends up with its own unique shape.

As a concrete example, let's look at Databricks Unity Catalog.
Unity Catalog's credential vending is a mechanism that issues short-lived credentials to external systems connecting via the Unity REST API and the Iceberg REST Catalog.

https://docs.databricks.com/aws/en/external-access/credential-vending

The "Credential Vending" section of Chapter 9, "Open Access," in the book Data Governance with Unity Catalog on Databricks explains that the principal accessing the data must first be granted the necessary read or write permissions, and that each vending API call is recorded in the audit system table as generateTemporaryTableCredential.

The credential vending entry point follows the REST Catalog spec, but both the permission model that decides "who gets which permissions" and the audit mechanism that records those calls are Unity Catalog's own implementation.
Here, too, you can see access control and auditing converging in the same place: the catalog.

Catalog-Specific Capabilities Create Lock-in

Organizing this structure from a migration perspective gives the following.

On auditing, the REST Catalog Protocol includes a mechanism where, if the server exposes a metrics endpoint, clients report scan and commit metrics to it.
See the "Metrics reporting" section below.

https://iceberg.apache.org/docs/nightly/rest-protocol/

In the sense that catalog operators can see how tables are being used, you could say part of auditing is starting to make its way into the spec.
However, it does not define the format of audit logs that record "who accessed what, and when" for governance purposes.

On views, Unity Catalog's official documentation states that while the Iceberg REST Catalog API exposes tables, it does not expose views, so external Iceberg engines cannot read views defined in Unity Catalog.

https://docs.databricks.com/aws/en/iceberg/

So there are cases where you can take your tables with you, but not the views you've built on top of them.

The parts the Iceberg spec covers are genuinely open.
The reason a catalog migration can be "just re-register the tables in the new catalog, without copying data" is precisely because the data and metadata layers are open.

Of course, there are cases where you can export lineage information to another system or feed audit logs into an external platform.
However, nothing in the Iceberg spec guarantees that you can move between catalogs while keeping the same semantics, the same permission structure, and the same operability.

What gets locked in, I think, is not so much the catalog itself as the management and governance machinery built on the assumption of that catalog.
If your permission settings, authorization model, auditing, lineage, sharing, and operational jobs depend on a specific catalog's features or representations, changing the catalog means revisiting all of them too.
And much of what we expect from a catalog in production is concentrated right here.

The Preconditions Behind "The Data Is Still in S3"

So far, I've been writing on the premise that "the data remains in storage, and since it's Iceberg, other engines can read it."
That premise, however, comes with conditions.

The first is that the storage is under your own control.
Unity Catalog's official documentation explicitly states that credential vending is not supported on workspaces that use default storage.

https://docs.databricks.com/aws/en/iceberg/

In setups where data lives in vendor-managed storage, the premise that "storage lock-in has been solved" may not hold in the first place.

The second is that the table is actually managed as Iceberg.

Unity Catalog's Iceberg REST Catalog provides read, write, and create access for managed Iceberg tables, but only read-only access for Delta Lake tables that have Iceberg reads enabled.

https://docs.databricks.com/aws/en/tables/managed

A table that is really Delta underneath and merely readable as Iceberg cannot be written to from outside, so the "◎ for data files and metadata" in the table above doesn't apply as-is.

When evaluating data portability, it seems necessary to check not just the format, but also "where the data lives" and "which metadata is the source of truth."

Why Does Everything Converge on the Catalog?

Why do capabilities like permissions, auditing, sharing, and lineage all end up concentrated in the catalog?

This isn't speculation; the official Iceberg documentation itself explains it.
The "Credential vending" section of the REST Catalog Protocol page states that because vended credentials are scoped to a table's storage prefixes, the catalog becomes the single point of access control.

https://iceberg.apache.org/docs/nightly/rest-protocol/

As mentioned earlier, with the REST Catalog the catalog also takes on commits.
That means both the place that finalizes the table's state and the place that grants access to storage are the catalog.

From here on this is my own interpretation, but if state management and access authorization both converge in one place, it's natural to record audits there too.
The same goes for data sharing and lineage, and as a result, governance capabilities concentrate in the catalog.

In other words, this concentration of capabilities in the catalog could be seen less as vendors trying to lock customers in, and more as a rational design that emerges from trying to centrally manage access to data.
And I feel that this very rationality is what ends up creating a new kind of switching cost.

On top of that, trying to standardize access control runs head-on into each cloud's IAM and identity platform.
Permission settings from one environment don't carry over to another because the way principals are represented is fundamentally different, and reducing them to a common format isn't easy.
I suspect this gap won't be closed anytime soon.

Does Using an OSS Catalog Avoid Lock-in?

Let's also think about OSS here.

Chapter 9, "Open Access," of Data Governance with Unity Catalog on Databricks, the book I mentioned earlier, explains that because Unity Catalog supports both the Unity REST API and the Iceberg REST API, you can migrate to other catalogs that support these formats, which eases vendor lock-in concerns.
The same chapter also introduces a setup that self-hosts the OSS version of Unity Catalog.

I think this is correct when it comes to data portability.
And if you keep using an OSS catalog, you may also be able to move to another cloud or another operator while keeping the same permission model.

But what's solved in that case is "lock-in to a vendor," not "lock-in to a specific catalog implementation."
When moving from one OSS catalog to another, or to a different managed catalog, you still need to revisit permissions, auditing, sharing, and operations.
It may be closer to reality to say that the lock-in target has simply shifted from "the vendor" to "the catalog implementation."

Portability

Based on all of this, I'd like to split the word "portability" into two.

Data Portability

Can the same data be read and written from a different environment, and can the data be migrated? This is what open table formats solved.
That said, it assumes the storage is under your control and the tables are managed as Iceberg.

Governance Portability

Can the same data be moved to another environment while keeping the same permission structure, the same auditing, the same sharing model, and the same operability?
This is the part that remains unsolved.

"Being able to take your data with you" and "being able to take the way your data is managed with you" are entirely different problems.
Looking only at the former and declaring "lock-in has been solved" seems like evaluating only half the picture.

I think it's closer to reality to say that lock-in hasn't disappeared, but that the layer where lock-in can occur has moved up.
As storage, table format, catalog, and governance separate into distinct layers, the center of gravity has shifted upward by exactly as much as the lower layers have opened up.

Choosing a Catalog Is as Weighty a Decision as Choosing an Engine

Engines also have their own SQL dialects, optimizations, UDFs, and operational features, and migrating between them is by no means easy.
But I feel that changing catalogs, which means rebuilding permission design, audit design, and operations all at once, is a qualitatively different kind of cost from an engine migration.
I think it's fair to treat choosing a catalog as a decision just as weighty as choosing an engine, and in some cases even more so.

That said, it's also true that most of the time you'll use your data platform's standard catalog. Or rather, if you want to make full use of a data platform's features, you're often effectively required to use that product's catalog.

Conclusion

This time, I sorted out the lock-in Iceberg has solved and the lock-in it hasn't, by separating things into layers.

To summarize:

• What the Table Spec opens up covers data files, metadata, and the concept of atomically swapping the pointer; how the pointer is kept is not standardized.
• The REST Catalog spec standardizes commits and access delegation such as credential vending as the catalog's role, and the official documentation itself positions the catalog as "the single point of access control."
• On the other hand, the permission model that decides "who gets which permissions," audit logs, lineage, and the sharing model are all outside the spec and specific to each catalog implementation.
• Data portability also has preconditions: the storage must be under your control, and the tables must be managed as Iceberg.
• Using an OSS catalog lets you avoid vendor lock-in, but lock-in to the catalog implementation remains.
• Lock-in hasn't disappeared; it seems closer to reality to say the layer where lock-in can occur has moved up, from storage and format to catalog and governance.

I think the term "open table format" does keep its promise when it comes to data and metadata.
On the other hand, the catalog and governance layers above them are not guaranteed the same portability.

"Being able to take your data out" and "being able to take your data's management and governance out" are separate problems.
That's exactly why, when evaluating a data platform, I think we need to look not only at the openness of storage and table formats, but also at the portability of the catalog and governance layers above them.

I hope this article is helpful to anyone thinking through catalog selection or governance design.