Apache Iceberg: the record you can take with you

A group acquires a company. The buyer runs its reporting on Snowflake; the acquired company kept its data in its own data centre and queried it with Trino. The integration plan has a line that says “migrate the data”, and everyone knows what that line costs: a year of copying, two sets of numbers in the meantime, and a dependency on whichever platform the copy lands in.

The alternative is for the record to be in a format both sides already read, with a catalogue the group controls. Apache Iceberg is, at the moment, the format on which the most rivals agree, and the reason is not sentiment. It is the shape of its metadata. Hurukan’s record is designed to be portable in exactly this sense, and the Data layer that holds it is in build.

A table as a tree, with one pointer

Iceberg describes a table as a tree of files that are never edited. At the top is a table metadata file; it points to a snapshot; the snapshot points to a manifest list; each manifest lists data files and delete files with their statistics. A change to the table writes new leaves and a new root, and then does one small thing: it swaps the pointer in the catalogue from the old root to the new one. The specification makes that swap atomic.

Two consequences follow. Every root that has existed is still a complete, readable version of the table, which is time travel, and in Hurukan it is the answer to “what did the agent see when it proposed this”. And several engines can write to the same table safely, because the only thing they contend for is the pointer.

The current release is Iceberg 1.12.0 from September 2026 (checked 21 October 2026). Version 3 of the specification is adopted: deletion vectors stored in Puffin files, row lineage, a VARIANT type, geometry types, nanosecond timestamps and default column values. Version 4 is in development and not adopted.

Hidden partitioning, and why an analyst never asks about it

Iceberg derives partition values from column transforms — the day of a timestamp, a bucket of an identifier — and records the transform in the metadata. A query that filters on the timestamp is pruned to the right partitions without the person writing it knowing that partitions exist. When the volume changes, the partition specification can evolve without rewriting the data; old files keep the old layout, new files get the new one, and both are read correctly.

For a record that is meant to be read by superintelligence rather than by people who know the storage layout, that property matters more than it sounds. The ontology can name a table “payments” and leave the layout to the Data layer.

The catalogue is where sovereignty sits

Everything above is files. Files can live on any object store, including one in the company’s own building. What the files cannot do is tell a reader which root is current and whether this person may read it. That is the catalogue’s job, and it is the place where a company either owns its record or rents it.

Iceberg publishes an open REST catalogue specification: namespaces, tables, views, commits, scan planning and credential vending behind one HTTP interface. Apache Polaris, an implementation of it, became a top-level Apache project in February 2026 and released 1.8.0 in September 2026 (checked 21 October 2026). Other implementations exist. The point is that a catalogue can be run on infrastructure the company chooses, and an engine from any vendor can talk to it.

That is how the security and sovereignty layer treats the record: permissions enforced at the catalogue, on the company’s own server, for people and agents alike. A vendor’s engine that reads the record does so with the credentials the catalogue hands it, and no more.

Who reads it

The list, from the vendors’ own documentation on 21 October 2026: Snowflake reads and writes Iceberg tables up to specification v3 and can use an external REST catalogue; Amazon S3 Tables store Iceberg natively, including v3; Google’s lakehouse catalogue exposes an Iceberg REST endpoint; Databricks offers managed Iceberg tables in Unity Catalog with a REST endpoint. Spark, Flink, Trino and DuckDB read it directly.

For the acquisition in the opening paragraph that list is the plan: one record, on the group’s storage, behind the group’s catalogue, read by Snowflake on one side and Trino on the other, with no migration line in the budget.

What agents do here, and what people keep

An agent can draft the job that evolves a partition specification when volume grows, the compaction that merges small files, the expiry of old snapshots once the retention the company has set is reached. Each is a proposal with a test result, approved by a person, recorded like any other change in Hurukan.

People keep the decisions that cannot be inferred: which catalogue, and where it runs; who may read which namespace; how long history is kept and when a person’s data is deleted for good; and whether to move a table to a newer specification version, since a reader that does not know v3 will not read a table that uses it. The question of which data may leave the building at all is a separate one, taken up in Which data must never leave the building.

A test you can run today

Ask who in your company could, this week, point a new engine at your most important table and have it read correctly without an export. If the answer involves a vendor’s support ticket, the record is not yet yours.