Apache Paimon: one state table for the operator, the planner and the agent

An assembly order changes state eleven times between release and shipment: planned, released, started at the first station, held for a missing part, resumed, inspected, one unit rejected, reworked, packed, confirmed, shipped. The operator at the station sees her step on a terminal in Operations and quality. The planner sees the order in a board. The operations agent, asked whether the week’s plan still holds, sees a report. Three views, and in most companies three tables, refreshed on three schedules, so that for an hour every afternoon the operator, the planner and the agent each believe a different thing about the same order.

The Data layer of Hurukan is designed so that there is one state table for that order, and everyone reads it. The pillar text explains why the live state is a separate memory from the record. This text is about the format that table is built in, Apache Paimon, and about what that choice costs. The layer is in build; this is how it is being designed.

A table that is updated, not appended to

Most table formats in a lakehouse assume data arrives and stays. New rows become new files; changing a row means rewriting a file or marking it deleted. That is the right design for a record, where nothing is ever changed. It is the wrong design for the state of an order that changes eleven times before it is done.

Apache Paimon takes a different shape. Its primary-key tables keep rows sorted by key inside buckets, in a log-structured merge tree: new versions arrive in a top level, and background compaction merges them down into sorted files, so the table stays compact however often the same keys are rewritten (Paimon table types, checked 21 October 2026). A reader sees the latest version of every key without scanning a history. Tables without a primary key are append tables, which fit the record side; here the primary-key kind is the point. The current release is 2.0.0 of 7 August 2026 (release catalog).

Merge engines are ontology rules

What makes Paimon more than a fast upsert table is that the format itself decides how a new version of a row combines with the old one. Paimon calls this the merge engine, and in the Data layer each choice corresponds to a rule the ontology already states about an object.

Deduplicate, the default: the newest version wins. The right rule for an order’s status — the latest event is the truth.

Partial update: each writer fills only its own columns. The order state row has the planned dates from planning, the actual start and stop from the station terminal, the inspection result from quality, the packed quantity from the warehouse. Four modules, one row, none of them overwriting the others. Sequence groups let each column set carry its own ordering, so a late message from the station cannot undo a later one from quality (partial update).

Aggregation: numbers add up across events — hours booked on an operation, units completed, units rejected.

First row: the first version is kept. Useful where the ontology says a value is set once, like the moment an order was released.

In the ontology these are described as properties of the object: which field is set once, which accumulates, which belongs to which module. Expressing them in the table definition means the rule holds for every writer, including an agent-built pipeline that did not exist when the rule was written.

One table, three readers, one changelog

The operator’s terminal, the planner’s board and the agent’s view are three readers of the same table, each filtered differently. Flink writes it from the module event stream and reads it back in lookup joins. Spark, Trino, Hive, StarRocks and Doris read it too (engine support), so a report engine and a notebook see the same row the terminal shows.

The table also emits its own changes. A changelog producer — input, lookup or full-compaction, chosen per table — turns every merged change into a stream. The operations agent subscribes to “order held” rather than scanning the board, and the planning view updates because the table changed, not because somebody pressed refresh.

For audit, Paimon keeps snapshots and lets a tag pin one permanently. When the agent proposed moving three orders to next week, the tag taken at that moment shows the board it was looking at. The proposal, the decision and the snapshot sit next to each other in the record.

What it costs

A Paimon state table is not free, and the Data layer is honest about the bill.

Compaction never stops. The merge tree is kept compact by background work that consumes compute continuously. The more often keys are rewritten and the fresher the table has to be, the more it costs. This is the reason not every table in the layer is a Paimon table: the record itself, which only grows, does not need it.

A streaming engine is a prerequisite. Writing state continuously means running Flink continuously, with someone responsible for it. Companies without that capability should start with tables refreshed in minutes by batch jobs and move to streams only where a decision needs seconds.

The key has to be right. A primary-key table is only as good as its key. If the ontology does not yet say what identifies an order across modules, the table cannot be built, and the honest first step is the ontology, not the format.

Agents help with the routine part: proposing bucket counts and compaction schedules, drafting the pipeline that maps module events onto the row, noticing when a table’s lag grows. People decide the key, the merge rule and how long snapshots are kept — because those are statements about what the company considers true.

A test you can run today

Take one object that several teams watch — an order, a job, a shipment — and ask each team to show you where they look it up. Count the screens. Then pick one field, such as the quantity completed, and ask each of them what it says right now.

If the answers differ, you have found the row that the Data layer is designed to keep once. If they agree only because somebody reconciles them every afternoon, you have found the person doing by hand what a merge engine does in the format.