How a deployment is built.
Every deployment goes through four phases: preparation, model, harness and finalization.
1 Preparation
We work with you to learn three things.
- Your hardware
- VRAM and compute. The agent is then targeted at that hardware and tested against it. The smallest machine we build for: one GPU with 16 GB of VRAM.
- Your use case
- Math, physics, chemistry, electrical engineering and computer science, and so on. It decides the toolkits and the agent's workflow.
- Your data
- Any data of yours the model could be specialized on.
2 Model
We start from a strong open-weight model that fits your hardware. It can then be specialized two ways.
- For our framework
- A QLoRA fine-tune that makes the model more dependable inside our agent framework: its interface, tools and workflow.
- For your field
- A fine-tune on domain-specific material.
We can also distill a stronger model into the smaller one, focused on your use case.
We are benchmarking open-weight models now, to know which to start from on which hardware. Fine-tuning and distillation are planned: this is how we will work, not something we have shipped.
Where a fine-tune runs, and whether your data leaves your machine for it: it runs on our own machine, not on a cloud service, so a fine-tune on your data means sending us that data, and one for our framework needs none of it.
- base
- 4096 x 4096 = 16,777,216, not trained
- adapter
- 2 x 4096 x 16 = 131,072, trained
- share
- 0.78 % of the layer
One layer, 4,096 wide, as an example. The blocks are drawn thicker than scale so they can be seen.
3 Harness
The harness is what turns a local model into a working agent. It has five parts, and it is the part of Herth that is built and working today.
Below is a small model of it that runs in your browser. Follow one unit of work round the loop and watch the record fill. Then change what the model proposes, and see the gate or the check stop it.
How the agent is kept off the network: the agent's process runs in its own network namespace with no interface to the outside, so the operating system refuses every connection it tries and the harness records the attempt as blocked.
- Context
- What the model is given at each step.
- Model
- Proposes the next step. On this page its turn is scripted.
- Gate
- Tool calls are checked before they execute.
- Tools
- Toolkits built for your use case.
- Verify
- Hallucination checks on what the model produces.
Context
What the model is given at each step.
Model
Proposes the next step. On this page its turn is scripted.
Gate
Tool calls are checked before they execute.
Tools
Toolkits built for your use case.
Verify
Hallucination checks on what the model produces.
Transparency
You can see what the agent did and how. Every part of the loop writes a row.
| Step | Part | What it did | Status |
|---|---|---|---|
| 01 | context | task, 8 values, 3 tools | ok |
| 02 | model | proposes mean(od600) | ok |
| 03 | gate | mean is on the list | ok |
| 04 | tools | mean(od600) = 0.4275 | ok |
| 05 | verify | model wrote 0.43, tool returned 0.4275 | ok |
| 06 | context | result added | ok |
| lap | 6 rows, 0 blocked | done |
A miniature of the loop, written for this page. No model runs here: what the model proposes is scripted, and you choose the script. The gate, the tool and the check are executed in your browser, on eight sample values.
4 Finalization
The model and the harness become one agent, on your machine.
- Quantize
- We quantize the model to fit your hardware. The figure works the sum for any model and any machine.
- Integrate
- The quantized model is integrated with the harness.
- Benchmark
- Then we benchmark the whole agent on that hardware: the model and the harness together, on the machine it will run on.
- levels
- 2^4 = 16
- rms error
- 1.85 % of the range
- weights
- 27e9 x 4 / 8 = 13.5 GB
- memory
- 24 GB: fits, 10.5 GB to spare
Weights only, and every weight at the same width. Real formats keep some weights at more bits, so a file labeled 4-bit averages more than 4 bits a weight and is larger than this sum. A running model also needs memory for the text it is working on. Leave room for both.
Bring Herth in
Tell us what you work on and what you would hand to an agent first. Every deployment starts with that conversation.