Kairolith

Whole-system performance for agentic hardware development

Your agents can build the system. Kairolith tells them how it performs.

Kairolith runs the whole thing, your hardware and your unmodified software on one clock, and answers the question that decides the design: where does the time go, across every component. The answers start arriving while the run is still going, and you or your agent decide when you have seen enough.

The problem

The metric nobody can test for.

Post-Moore computing wins through radical redesign: new splits of the work between hardware and software. That makes throughput and latency properties of the complete system, not of any part. The path a real request takes crosses every component you own and several you do not. That path is what you were asked to make fast, and it is the metric that decides whether the design is worth building.

Coding agents have reached this work. They produce the driver, the firmware and the RTL, and they pass every test each piece has. Then the pieces meet, and the loop stalls at the point where it should get interesting: the agent can change anything and cannot see what the change did to the whole.

One request crossing host software, NIC, wire, NIC and host software host0/apps/client · kernel nic0 switch0 · wire nic1 host1 · driver · softirq · app syscall doorbell descriptor fetch · DMA wire DMA · interrupt softirq · wake-up 12 µs
Dashed lanes are components you own.
unit test

Checks one block. The request never enters it.

testbench

Has no driver in it. Doorbell, interrupt and wake-up timing is whatever its author guessed.

benchmark

Returns a number with no reason attached.

profiler

Sees one host and stops at the device boundary.

What you give Kairolith

A dozen lines, your software, and the models you already have.

Your system

Hosts, devices, links and the applications that run on them, described in a few lines. Two hosts, two NICs and a switch is a dozen lines.

Your software

The kernel, drivers, firmware and applications, unmodified. Linux is understood out of the box; your driver on top of it needs nothing extra.

Your models

The simulation models you already have for your own components: RTL through Verilator, SystemC, or a behavioral model in C++, attached through a lightweight integration adapter. Everything you do not build yourself, hosts, network, off-the-shelf devices, comes from existing simulators that Kairolith composes for you.

ping2 · description → systemillustrative syntax
1system ping2
2 host0 linux apps: [client]
3 host1 linux apps: [server]
4 device nic: rtl(verilator, ./nic)
5 nic0 nic pcie: host0
6 nic1 nic pcie: host1
7 switch0 ports: 2
8 link nic0 ↔ switch0.port[0]
9 link nic1 ↔ switch0.port[1]
10 phase echo: client → server, 100×
host0 linux · client host1 linux · server nic0 rtl · verilator nic1 rtl · verilator switch0 2 ports · provided port[0] port[1] provenance · 1 line per run: host*=timing cpu · nic*=your rtl · switch0=behavioral
Every name in the description becomes an address the agent can query: host0, nic1, switch0.port[1], host0/apps/client.

What you get back

Where the time goes, across the whole system.

For any request, the full path across hardware and software with the time attributed to each segment, and the gaps labelled. The same view for the tail: the slowest one percent, what they have in common, and which component they spend their extra time in.

ping2 · one round trip, attributed to the component that spent it run 41 · 5 of 100 round trips · answers so far
pcie eth eth pcie host0 apps/client kernel · your driver linux · unmodified nic0 your nic tx · rx · irq rtl switch0 port[0] · port[1] wire nic1 your nic rx · irq · tx rtl host1 apps/server kernel · your driver linux · unmodified cause: nic1 interrupt throttle client sw · 41 irq→xmit · 34 … reply path · 78 client sw · 21 nic0 tx · 3 wire · 1 0 50 100 150 178 µs
host0 software nic0 switch0 · wire nic1 host1 software gap: waiting for interrupt · 12 µs your model
Hover a component or a segment to follow the mapping. The gap is time host1 spent waiting; the cause is in nic1. Every segment is a handle the agent can open. The reply-path split is illustrative.

The same request, as the agent receives it

Every number is a handle

The agent opens one level at a time, component, domain, operation, event, and never receives ten thousand spans by accident. It stops descending when it has the answer: here, three levels down from a 178 µs round trip to the throttle timer that held the interrupt back.

The whole answer fits in a few hundred tokens, in the vocabulary of the system you described, so the agent spends its context on the design instead of on the trace.

annot:rtt · phase echo · one request, opened three levelslevels: component → operation → event
annot:rtt178request
host0/apps/client · sw41 + 21component
nic0 · tx3component
switch0 · wire1component
host1 · irq→xmit gap: waiting for interrupt 1234component
nic1 rx.dma.write done → irq raised12.0operation
itr timer expiry (interrupt throttle)event
… reply path78component
tail view: p99 · 181 µs · same split as the first five · no component stands out≈ 300 tokens
Highlighted rows are the ones the agent opened in the session below. The tail line is illustrative.

How it works

Every component on one clock. Every event, linked.

host0 nic0 switch0 nic1 one simulated clock
Events recorded in simulated time, linked across host, device and network.

A virtual prototype of the whole thing

Your hardware as a behavioral model early or as RTL later, your real software stack on top, every component on one synchronized clock. Every event in every component is recorded in simulated time, so recording costs the measured system nothing, and the events are causally linked across host, device and network. That linked store is what the agent queries.

agent asks nic1 rx.dma → irq answer nic1 · 12.0 µs the names from your description never in the answers full-system verilator device model network provenance, 1 line per run: host*=timing cpu · nic*=your rtl · switch0=behavioral
nic1 is your name from the config, in the question and in the answer.

Your vocabulary in, your vocabulary out

The agent asks questions in the terms of the system it is building, and the answers come back in the same terms. Which model backs each component is one provenance line per run, so the agent knows whether it is optimizing your design or an artifact of how it was measured. The simulators underneath never show up in the answers.

A custom kernel or an RTOS is described once, as a profile, and from then on its software stages are named the same way, in every iteration.

1:40 · driver probed boot · probe 3:12 · 5 round trips — answers now + rtt 1…5 40:00 · phase complete + tail every query answers on what has landed so far
Queries never wait for the end of the run.

Answers accumulate as the run proceeds

A run is not a job that returns a result at the end. It is a feed. Events land in the store as the simulation produces them, and each one is linked to the request it belongs to on arrival. So a query at three minutes is not a partial file being parsed, it is a smaller version of the same answer: five round trips attributed instead of a hundred. The run keeps going, the store keeps filling, and the same question asked later simply has more behind it, including the tail.

A session

Ask it to be faster. Watch it find the 12 µs.

claude code · kairolithping2 · runs 41 → 42
you
the round trip is 178 µs. make it faster.
agent
run_submit(system=ping2, expectations=[...])           → run 41
run_wait(until="milestone:driver_probed")              → 1m 40s wall
run_wait(until="trace:annot:rtt count>=5")             → 3m 12s wall, run continues
trace_breakdown(root="annot:rtt", phase="echo")        (5 of 100 round trips so far)
  client sw 41 · nic0 tx 3 · wire 1 · host1 irq→xmit 34 · … · client sw 21
  gap: waiting for interrupt 12 µs
trace_get(id=…, level="operation", component="nic1")
  rx.dma.write done → irq raised: 12.0 µs   (interrupt throttle timer)
agent
five round trips are enough to see where the time goes. leaving 41 running for the tail, editing now.
lowers the rx interrupt throttle default in your driver, resubmits  → run 42
run_wait(until="trace:annot:rtt count>=5")             → 3m 09s wall
compare(a=41, b=42)
  inputs: driver image changed; hardware, config: identical
  timing: host1 irq→xmit 34 → 22 µs; first divergence nic1, irq 12 µs earlier
  structure: identical; register traffic: identical
  expectations: 3/3 pass so far
agent
12 µs of the 178 was nic1 holding the interrupt back for moderation. host1's software is unchanged. The same setting on nic0 should take another 12 off the reply. Run 41 finished meanwhile: the tail matches the first five, nothing new there.

Illustrative transcript. The tool names and the first-run timings are real; the phase length and the second-run numbers are not.

Feedback while it runs

Milestones arrive as the system reaches them, and every query answers on whatever has happened so far. A run that takes forty minutes to finish reports on the interesting phase in three, and the agent decides when it has seen enough: keep it running for the tail, or stop it and spend the compute on the next iteration.

Expectations declared at submit are checked live against the feed, and a violation stops the run on the spot.

phase:echo:rtt.p99 < 200us
nic0 tx.doorbell → wire < 3us
no console:/Oops|panic/
on_fail: kill
wall clock · runs 41 and 42 overlap0 → 10 min shown · run 41 finishes at ~40 min
0:00 2:00 4:00 6:00 8:00 10:00 → run 41 submitted driver probed first packet on wire rtt count ≥ 5 … tail · finishes ~40:00 · matches the first five 3:12 · seen enough → edit driver run 42 submitted rtt count ≥ 5 · compare(41, 42) expectations 3/3 · live
Run 42 is submitted while 41 is still simulating its tail.

What changed, exactly

Compare this iteration against the last one, or one request against the rest of its phase. Inputs first, so the agent learns that the driver changed and the hardware did not before it sees a timing delta. Then structure, register traffic, timing, critical path and batching, ranked, with the earliest divergence as the headline.

Runs are reproducible, so there is no noise floor: every difference is real, and the earliest divergence tells you where to look. Mark a baseline per system and every new iteration comes back as a regression report: regressed, improved, new behavior.

compare(a=41, b=42) · same scaleinputs: driver image changed · hardware, config identical
run 41 baseline 34 178 run 42 candidate 22 166 −12 µs first divergence · nic1 · irq raised 12 µs earlier structure identical register traffic identical timing host1 irq→xmit 34 → 22 µs expectations 3/3 pass so far
The hatched region is the 12 µs interrupt wait in the baseline; the dashed outline is where it used to be. Second-run numbers are illustrative.

The loop

Hardware development becomes a software loop. Long before there is silicon.

Every change to a driver, a firmware image, a device model or an RTL block runs as part of the whole system and starts reporting, attributed and comparable, while the change can still be made.

The iteration is as short as the evidence needed, not as long as the run: an agent stops reading when it knows enough and stops the run when it will learn nothing more. Agents run that loop on their own, driven by a developer's question, or on every commit, and the architect gets a ranked list of what moved the number and why.

Start

One link. Everything else happens from there.

Read https://[host]/agent.md and follow it.

Paste it into Claude Code, Codex or whatever runs your agent. The file tells the agent how to connect, which questions to ask in which order, how to follow a run as it happens instead of waiting for it, and when it has seen enough to act. Same API behind a CLI for humans.

Claude CodeCodexCLI