Small models on your own hardware: when they are enough

A company wants to sort incoming customer emails into a fixed set of categories and pull out order numbers. It sends every email to a large external model. It works. Then the bill grows with the volume, each email waits a few seconds for an answer from another continent, and the legal team asks why customer correspondence is leaving the company at all.

Someone asks whether a small model on a server in the office could do the same job. Nobody knows, because the choice of model was never measured. It was made once, for the whole company, on reputation.

The cause is that model choice is treated as a platform decision rather than a per-task one. Frontier models are general: they are built to do almost anything well. Most of what a company needs from a language model is narrow, repetitive and well defined. For narrow tasks, a much smaller open-weight model, possibly adapted to the task and compressed to run on modest hardware, may be as accurate, faster, cheaper per call and entirely under the company’s control. Or it may not be. That is what an evaluation is for.

Where small models are often enough

Classification and routing. Sorting inputs into a fixed set of categories, priorities or teams.

Extraction. Pulling fields out of documents with a known shape: invoice numbers, dates, amounts, order references.

Redaction before anything leaves. A small local model removes personal and confidential details, so that a larger external model can do the rest of the work on what remains.

Embeddings for internal search. Turning internal documents into vectors for search does not need a frontier model, and the documents never have to leave.

Speech recognition on site. Transcribing speech locally keeps staff voices inside the company, which matters for voice in hands-busy work.

Short, templated drafting. Standard replies whose structure and tone are fixed.

Where they fall short

Long, multi-step reasoning across many sources and many turns.

Open-ended writing for customers and partners, where quality is judged by people outside the company.

Languages and domains with little training data. Performance in a less common language or a specialised field has to be tested, not assumed.

Writing changes to the system itself. Generating code and configuration for review is where the most capable models earn their cost.

Anything you cannot evaluate. If you cannot tell whether the answer is right, you cannot tell whether a small model is good enough.

The costs on your side

A model you run is a system you operate: hardware, updates, monitoring, security patches and someone who owns it when it misbehaves. Open weights also come with licences, and these differ; some carry conditions on commercial use. Read the licence of each model before it goes anywhere near production.

How to decide

Decide per task, with evidence. Build an evaluation set from real examples, covering the common cases and the awkward ones, with correct answers checked by the people who do the work. Run the frontier model you use today and one or two small candidates over it. Compare accuracy on the cases that matter, response time, cost per thousand calls, and where the data goes. Choose the smallest model that meets the bar, and run the evaluation again when either model changes.

Hurukan is designed to make that choice routine. The Super Intelligence layer routes each task to a model under the company’s rules, between frontier models and models running locally, and a model can be replaced without touching the agents that use it. The Security and Sovereignty layer is built for infrastructure the company owns or chooses. Both are in build. Which data should go to a local model in the first place is the subject of which data must never leave the building.

A test you can run today

Pick one narrow task you currently send to an external model: classifying, extracting or tagging. Collect a sample of real inputs from last week and have the person who normally does the job write down the correct output for each.

Run the same inputs through a small open-weight model on a workstation, using one of the common tools for running models locally. Compare its outputs, and those of the external model, against the correct ones.

If the small model matches on the cases that matter, you have found a task that can stay in the building. If it does not, you now have an evaluation set, which is worth keeping: it will tell you the day a small model becomes good enough.