The audit trail is a table: Delta Lake, Iceberg or Hudi

An auditor asks a simple question about a payment approved in March: what did the person who approved it see on the screen, and what had the agent that drafted it read before proposing it? In most companies the answer is a reconstruction. Somebody pulls a backup, somebody else searches an inbox, and the result is an informed guess about a state that no longer exists.

In Hurukan every agent proposal and every human decision is an event in the company’s record, and the record is designed so that this question has a plain answer: a version of the tables as they stood at that moment. That property does not come from the files. It comes from the table format on top of them. This text is about what the three open table formats do with the question, and how to choose between them. The Data layer is in build; what follows is how it is being designed. The reasoning for having a record at all is in the live state and the record.

What an audit trail needs from a table

Files in the record are never edited. A change writes new files and a new version of the table’s metadata that says which files now count. Three things follow for the audit.

Every version is addressable. “The table as of 14:02 on 12 March” must be a valid query, not a restore job. In the formats below this is time travel, and it is the mechanism behind “what did the agent see”.

Every version has an author. The metadata of a change has to carry who or what made it: a person, an agent acting on a person’s approval, or a system task such as compaction. Hurukan records the actor in its own decision log, but the table version is the proof that the log matches the data.

Old versions expire on purpose, not by accident. Keeping every version forever is a cost; removing them is a decision about retention and, for personal data, about deletion. That decision belongs to a person, and the table format has to make it explicit.

Three ways of keeping the plan

Each format keeps a plan of which files make up the table. The plan is where the audit lives.

Delta Lake keeps a log: a directory _delta_log/ with one JSON file per commit, named by a twenty-digit version number, each listing actions such as add, remove, metaData and protocol, with periodic Parquet checkpoints so a reader does not replay the whole history. The protocol defines it; the current release is 4.4.0 (checked 21 October 2026). Log retention defaults to 30 days and VACUUM to 7 days, which means time travel reaches back only as far as those two settings allow.

Apache Iceberg keeps a tree: a table metadata file points to a snapshot, the snapshot to a manifest list, the manifests to data and delete files with their statistics. A commit is an atomic swap of the pointer to the metadata file, and the swap happens in the catalogue. The specification is at version 3, with deletion vectors, row lineage and the VARIANT type; the current release is 1.12.0 (checked 21 October 2026).

Apache Hudi keeps a timeline: every action on the table — commit, compaction, clean, rollback, restore — is an instant with a state, and the history of the timeline is archived in an LSM structure. Records have keys and an index, so a change to one customer touches one file group rather than a partition. Tables are Copy-on-Write or Merge-on-Read, and release 1.2.1 added table version 9 (checked 21 October 2026). The timeline documentation describes the states.

All three answer the auditor. The difference is where the answer is kept and who else can read it.

Choose by catalogue, not by feature list

The feature lists have converged: transactions, time travel, schema evolution, row-level deletes and streaming writes exist in each. What has not converged is the place where the current version of a table is declared, and that place is where sovereignty is decided.

For Delta Lake it is the log directory itself, so any engine that can read the storage can find the current version. In practice the catalogue around it is Unity Catalog or, in Microsoft Fabric, OneLake, which stores its tables as Delta Parquet according to Microsoft’s documentation.

For Iceberg it is the catalogue pointer, and the catalogue speaks an open REST specification. Apache Polaris, a top-level project since February 2026 with release 1.8.0, implements it; Snowflake, AWS S3 Tables, Google’s lakehouse catalogue and Databricks all read and write Iceberg tables through it (checked 21 October 2026).

For Hudi it is the timeline on storage, read natively by Spark, Flink, Trino and Presto; other platforms generally reach a Hudi table through metadata translation.

So the question for a company is not “which format has the best audit”. It is: on whose infrastructure does the catalogue run, which engines must read the record, and can the record be taken along if the vendor changes? Hurukan’s answer is that the catalogue and the files run on infrastructure the company chooses, as described under security and sovereignty, and that the format follows from the engines the company already has rather than the other way round.

What stays with people

Agents are good at the mechanical part of the Data layer: drafting the compaction schedule, writing the job that expires old snapshots, proposing a partition change when queries slow down. Each of those is a proposal in the approval inbox, with the reasoning attached, and each is recorded when approved. Three decisions are never delegated.

Retention and deletion. How far back time travel reaches is the same setting as how long personal data survives a deletion request. A person sets it per table and signs it.

Protocol upgrades. Enabling a newer table feature can lock out an older engine that still reads the record. The gain is real; so is the reader that goes dark. A person weighs it.

The format itself. Changing it later is possible, as the next text in this series on XTable and UniForm explains, but it is a project, not a setting.

A test you can run today

Take one decision from last quarter that an auditor could plausibly ask about: a credit limit raised, a supplier approved, a discount granted. Write down, from your systems alone, what the decision-maker saw at that moment. Count the systems you had to open and the places where you had to guess. That count is the distance between your current record and one whose audit trail is a table.