--- title: "Apache Parquet: how the record is laid out so superintelligence can read the whole company" url: "https://hurukan.com/blog/apache-parquet-how-the-record-is-laid-out-so-superintelligence-can-read-the-whole-company" description: "Superintelligence reads the record constantly, on the company's own hardware. Parquet's layout sets what each question costs: row groups, statistics, codecs." --- # Apache Parquet: how the record is laid out so superintelligence can read the whole company September 18, 2026·4 min read·Editorial team A small on-premise server room seen from the doorway, two racks with a thin line of blue status lights, an engineer from behind holding a tablet that shows abstract blue-and-grey bar shapes, cool overhead light and a clean concrete floor A planning agent is asked, at seven in the morning, which open orders are at risk because their items have a quality hold. The answer needs three fields from the order record, two from the inspection record and one from the item master, across everything open in the company. It needs them before the planner sits down, and it needs them again at nine when the plan changes. Nobody notices the question when it is cheap. Everybody notices when the agents that ask it are the reason the servers were upgraded. Superintelligence reads the record far more often than people do, and in a sovereign system it does so on hardware the company pays for. The layout of the files that hold the record therefore decides something that looks like a performance detail and is in fact a cost of running the company. In Hurukan’s [Data layer](https://hurukan.com/system/data), which is in build, the record is designed to be stored in Apache Parquet. This text is about what that choice buys and what it demands. ## What is inside a Parquet file [Parquet](https://parquet.apache.org/docs/file-format/) stores a table column by column. A file holds a number of row groups; each row group holds one chunk for each column; each chunk is a sequence of pages. The metadata, including the schema and the position of every chunk, is written at the end, so a writer produces the file in one pass and a reader opens the footer first and then goes straight to the bytes it needs. The specification is at version 2.14.0 ([project blog](https://parquet.apache.org/blog/), checked 21 October 2026). The order of the questions a reader can skip is what matters for cost: **Columns it did not ask for.** The planning agent wants six fields; the file may hold sixty. The other fifty-four are never read. **Row groups that cannot contain the answer.** Each column chunk carries statistics, the minimum and maximum value and the count of nulls. If the question is about one item family and the file is sorted by item, most row groups are excluded from the footer alone. **Pages inside a chunk.** The [page index](https://parquet.apache.org/docs/file-format/pageindex/) records minimum and maximum per page and where each page starts, so even inside a chosen row group the reader jumps to the pages that match. **Values that do not exist.** A [Bloom filter](https://parquet.apache.org/docs/file-format/bloomfilter/) per column chunk answers “is this order number in here at all” without reading the chunk, which is what a point lookup on a high-cardinality field needs. What is left is read, and what is read is small, because columns compress well: repeated values become dictionary entries, runs become counts, and a general codec finishes the job. The [encodings](https://parquet.apache.org/docs/file-format/data-pages/encodings/) and [codecs](https://parquet.apache.org/docs/file-format/data-pages/compression/) are specified, with Snappy, GZIP, ZSTD and Brotli among the codecs and the newer LZ4\_RAW replacing the older LZ4 framing. ## Why the layout is a sovereignty question A managed platform hides the cost of a badly laid out record behind a monthly invoice. On the company’s own infrastructure, which is where a sovereign operating system lives, the cost is visible as processors, disks and the time an agent waits. The same question asked against a record sorted by the field it filters on, with row groups of a sensible size and dictionary encoding where values repeat, touches a fraction of the bytes it would otherwise touch. The decision about layout is a decision about how many questions the company can afford to let its agents ask. The wider design, including why the record is separate from the modules’ live state, is in [the live state and the record](https://hurukan.com/blog/the-live-state-and-the-record-two-memories-of-a-company-operating-system). Parquet is also the format every engine reads. Spark, Trino, DuckDB, Snowflake, BigQuery, Microsoft Fabric and the analysis libraries all open it natively. A record stored in Parquet is a record another engine can read tomorrow, which is one half of an exit; the other half, the table metadata above the files, is the subject of [the audit trail is a table](https://hurukan.com/blog/the-audit-trail-is-a-table-delta-lake-iceberg-or-hudi). ## Types the record will need The specification has grown towards the shapes a company record actually has. Version 2.11 added the VARIANT type for semi-structured values, the half-formed attributes that do not fit a fixed column, and GEOMETRY and GEOGRAPHY for locations; 2.12 finalised how VARIANT values may be shredded into typed columns for faster reading; 2.14 added a FILE type ([release notes on the project blog](https://parquet.apache.org/blog/), type definitions in [LogicalTypes](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md), checked 21 October 2026). For a record that has to hold a delivery address, a flexible set of inspection attributes and a reference to a scanned document on the same row, those are not exotic. What a given engine reads of them has to be checked per engine, and the design assumes that check is done before a type is used, not after. ## What agents propose, what people decide An agent that watches the questions asked of the record is well placed to improve its layout: it can see that most filters are on item family, that a table’s files have become small and numerous since a connector started writing every minute, or that one column has a dictionary that no longer fits. In Hurukan’s design those observations become proposals, a sort order, a compaction plan, a codec change, with the evidence attached, and the proposal is recorded whether or not it is accepted. What an agent does not decide is how long the record keeps each kind of row, which rows leave it when a person asks for their data to be removed, and whether a rewrite of a year of files, which compaction and re-sorting both are, is worth the processor time this week. Those are decisions about obligation and cost, taken by a person and written into the record like any other. ## A test you can run today Take the one report or model in your company that reads the most history. Find out two things: in what format its source files are stored, and whether anyone has ever sorted them by anything. If the answer is CSV or a database export, every question reads every byte. If the files are columnar but unsorted and never compacted, the first improvement is cheap and nobody owns it. The [Data layer](https://hurukan.com/system/data) page describes where the record sits in Hurukan; the [overview](https://hurukan.com/) shows what reads it. - Apache Parquet - the record - columnar storage - cost of reading - sovereign infrastructure ## Keep reading - [One identity for people and agents: who may do what](https://hurukan.com/blog/one-identity-for-people-and-agents) - [The live state and the record: two memories of a company operating system](https://hurukan.com/blog/the-live-state-and-the-record-two-memories-of-a-company-operating-system)