Apache Hudi: the coexistence phase, change data from the old ERP into the record

No company switches its operating system on a Monday morning. The order book, the stock, the open invoices and twenty years of customer history live in an ERP that will keep running while the first modules take over one process at a time. For months, sometimes years, two systems write to the same business, and the record has to stay true to both.

That is the coexistence phase, and it has a specific technical shape. The old system keeps changing rows; the record needs those changes in minutes, not in a nightly export; and it needs them as changes, so that a correction to one invoice does not mean reloading a year. Apache Hudi was built around that shape. The Data layer in Hurukan is in build; this text is about how the coexistence path is being designed and where Hudi fits in it. Why the record is separate from the live state is in the live state and the record.

What coexistence demands

Changes, not snapshots. An ERP changes thousands of rows an hour: a status, a payment, a corrected address. A nightly copy of every table is both too slow and too expensive. The record needs the row that changed, with what it was before and after.

Updates by key. The same customer, order or invoice arrives many times with new values. The record has to find the previous version of that one key and replace it, without rewriting the partition it sits in.

Incremental reading. Everything downstream — the ontology’s view of a customer, an agent preparing a credit decision, a report — should process what changed since it last ran, not the whole history again.

Deletion by key. A deletion request during coexistence has to reach the record as surely as it reaches the ERP, and be provably complete after cleaning.

How the change arrives

The ERP’s database already writes a log of its own changes for its own recovery. Change data capture reads that log and publishes each change as an event. In the design Hurukan is building, that event carries the contract the series describes under Apache Avro, so the consumer knows the shape of the row before and after.

From there Hudi Streamer, the project’s ingestion tool, reads from Kafka or directly from Debezium change events, in a continuous mode, with a schema registry, and writes the table (checked 21 October 2026). No code is written for the common case; the configuration is the design, and in Hurukan it is a proposal an agent drafts and a person approves.

What Hudi does with it

Hudi gives every record a key and keeps an index of where that key lives, so an update touches the file group holding that key rather than a partition. The index documentation lists the options kept in the table’s metadata: Bloom filters, a record-level index that can be global, secondary indexes and expression indexes added in release 1.0.0 (checked 21 October 2026).

Two table types answer two needs described under table types. Copy-on-Write rewrites the base Parquet file on every change; reading is simple and fast. Merge-on-Read appends changes to log files beside the base file and merges them on read; compaction folds them into new base files in the background. During coexistence the ERP tables are usually Merge-on-Read, because the writes never stop; a reporting table that changes once a day can be Copy-on-Write.

Every action lands on the table’s timeline — commit, delta commit, compaction, clean, rollback — with a requested, in-flight and completed state, and the history is archived in an LSM structure. Release 1.2.1 of September 2026 is current; 1.1.0 and 1.0.0 before it rebuilt the timeline and the indexes (checked 21 October 2026).

Three consequences matter for the record. Queries can be incremental: give me what changed since this instant, as the latest state or as before-and-after images. Several writers can share a Merge-on-Read table under non-blocking concurrency control, which the concurrency documentation limits to specific index types and not yet to clustering. And a deletion by key is an ordinary write, final after the cleaner removes the old file versions.

What people decide

The key. Which combination of fields identifies a row in the old system, and whether it stays stable across the migration. A wrong key duplicates customers for the life of the record.

The table type and the compaction budget. Merge-on-Read keeps the write latency low and moves the cost to compaction, which runs continuously and is paid for continuously. An agent will propose the schedule; a person signs the cost.

Retention and the cleaner. How many file versions to keep is the same decision as how far back “what did the agent see” reaches, and as when a deleted person is gone. It is set per table and recorded.

Where the engines are. Hudi is read natively by Spark, Flink, Trino and Presto. A company whose other tools expect Iceberg or Delta will reach a Hudi table through metadata translation, with the limits the series describes for XTable and UniForm. That is a reason to keep Hudi where it earns its place — the coexistence path — rather than everywhere.

The Data layer describes the connectors and the one data model that this path feeds. When the last process has moved to a module, the ERP stops writing, the change stream ends, and the record carries on without noticing.

A test you can run today

Pick one table in your ERP that changes all day: order lines or payments. Ask how it reaches your reporting today and write down three numbers: how old the copy is when someone reads it, how much of the table is reloaded to apply one day of changes, and how long a deletion request takes to be provably complete there. Those three numbers are what the coexistence path has to beat.