The Instrument Log | Bulkhead τ

The harness doesn't just decide outcomes. It decides what you're able to believe about outcomes.

What this is

Seventeen faults, caught over roughly a week of local-model benchmarking across two RTX 3090s, an RTX 2080 and a Ryzen AI MAX 390. Not one of them was a model behaving badly. Twenty more have been caught since, listed at the end. The ratio did not change. Every one was the instrument, the daemon, the log parser, or my own code.

They are collected here because of a property they share: each one produced a number rather than an error. A crashed run announces itself. A daemon quietly using two cards while the row says "solo" does not. It produces a clean, plausible, publishable figure that happens to be false, and nothing downstream can tell the difference.

That is the argument for the log. Anyone can report what their system did. The question worth asking is how many of these were sitting inside the run that produced the report.

Upstream and tooling

Five faults in things I did not write. These are the expensive ones, because there is no commit to review.

What broke	What it looked like	What it was
`CUDA_VISIBLE_DEVICES` does not hide a card from Vulkan	A valid single-card measurement	Two cards. The daemon enumerated `Vulkan0` as the other GPU and allocated on it. One config died on "failed to allocate Vulkan0 buffer". Requires `OLLAMA_LLM_LIBRARY=cuda_v13` as well.
`utilization.gpu`	"One card computes while the other idles, so layer-split is a pipeline"	The idle card was drawing 234 W against a measured idle baseline of 26-37 W, and climbing 70 to 78 degrees C while its reported utilization fell from 5% to 1%. Both cards were working. Power is the signal that holds.
`pcie.link.gen`	Flaky hardware: a slot that "flipped gen1 x16 to gen3 x16 in one day"	Link power management. The gen downshifts at idle and negotiates up under load, so it means nothing unless sampled during generation. Width is topology and is stable.
`ollama ps`	Model footprint, placement and context depth	Reported 29 GB against 34,728 MiB actually allocated. Reported "100% GPU" for a runner with a layer explicitly held back on CPU. Reported a 262,144 context that was not allocated at that depth.
`load_duration`	Cold-load cost	0.001 s on one daemon and 0.52 s on another, for the same already-resident model. A reporting difference between versions, not a load.

My own code

Nine faults in code I wrote or maintain. More than the upstream count, which is the honest ratio and the reason this page exists.

What broke	What it looked like	What it was
Layer capture parsed `journalctl -u ollama`	`status: ok`, no placement concern	A hand-started daemon is not under that systemd unit, so its placement was never visible. A configuration with a fifth of the model on CPU recorded as a clean pass. Only a VRAM shortfall and a 5x throughput drop gave it away.
Eviction check used bare `ollama`	Checking the daemon under test	Always talked to :11434 regardless of which daemon was being measured, so a solo run's eviction check inspected the wrong process entirely.
Dashboard glob card	One card failing to render	An unguarded `relative_to` on a path outside the repo took down the entire dashboard render, not just that card, in a generator whose own documentation says cards must degrade to an explicit empty state.
Hardware snapshot captured `power_limit`	Provenance	A static 320.0 on every row forever. The field that turned out to be the reliable activity signal, `power.draw`, was not captured at all.
Ledger writes inside the wall-clock timer	A 25% difference between daemon versions, 11-12 s against 15-16 s	Bookkeeping overhead. Two correction sections were written attributing it to the ollama version before anyone opened the traces.
`prompt_s_median` stored to one decimal	"Prefill is dead flat at 0.872 ms/token across three context depths"	Flat within the recorded precision. All three depths logged 11.2 s, so the identical figures were partly a rounding artifact. The conclusion survived. The phrasing did not.
Bench task authored out of scope	A model failing a task	The task reached outside its own fixture, so the failure was the bench.
Token usage discarded on every turn	"This endpoint does not return usage, so subscription spend is unmeasurable"	The endpoint returns it richly, including reasoning tokens and a per-message attribution breakdown. The client parsed it correctly and the caller bound it to a variable it never read again. Every lap since the file was written was unpriceable for that reason alone.
State-block generator was not idempotent	A working tool	The document gained a newline on every regeneration. Caught by the tool's own `--check` mode on its first run, which is the only reason it is a line in this table instead of a slow corruption.
`localhost:11434` is not this machine	A normal benchmark run on the 8 GB card in this box	An `ssh -L 11434:127.0.0.1:11434` tunnel to the other machine held `[::1]:11434`, and `localhost` resolves to `::1` first. The runner and the agent config both hardcoded `localhost`, so the harness reached the other host's daemon. The same model tag exists on both and both report the same ollama version, so a misrouted run loads cleanly and attributes a 24 GB card's work to an 8 GB one. The machine label is set by hand and witnesses nothing. Only surfaced because a separate mistake crashed the run first.
A hardware spec carried from a three-month-old finding	A controlled comparison: "same RAM, only the CPU differs"	The RAM had physically moved. The other machine ran the reference benchmark with ~15 GB and has 32 GB now, because the memory was transferred with the cards the night before. Reading its RAM live and calling that the reference condition compared the new machine against its own old numbers. The variable the experiment was built to isolate was never controlled, and the correction that followed was wrong in the opposite direction.
`offloaded 31/31 layers to GPU` on an MoE model	The model fully resident on an 8 GB card, published as a residency-verified row for weeks	The layer counter counts layers, not experts. A 17.33 GB MoE blob needs ~16.7 GB of VRAM; the card had ~7.4 GB free. Ollama resolved the 11.5 GB shortfall by moving every expert tensor to system RAM, kept dense weights on the device, and logged 31/31 anyway. Ten consecutive loads, every one the same. The rate was real but mixed-placement and host-memory dependent: 32-34 tok/s with the page cache warm, 4.0 tok/s with the host under pressure, ~580 major faults per token at 38% iowait and zero swap. Only the load-time fitting decision or a per-device VRAM measurement establishes MoE residency.

The one that was not a tool

Five consecutive correction sections were written into a working handoff about a single wall-clock discrepancy. Four of the five were wrong in whole or in part. The only correct one was the only one that opened a fixture before writing.

The first asserted an attribution from reasoning. The second inherited that premise without checking and turned it into a standing instruction about which figures to cite. The third read the traces and found the arms inverted. The fourth confirmed the third and caught two further defects. A fifth then retracted the third's mechanism as well.

The measurements were never the problem. The fixtures were on disk the entire time. What failed was documents asserting things about artifacts nobody had reopened, and each one made the next reader more confident. "Correction #3 says X" is not evidence. It is a claim with exactly the standing of the one being corrected.

The earliest one, and it has its own paper

This log starts in September because that is when I started counting. The oldest entry is older and was written up properly at the time: subprocess capture embedded VT100 terminal sequences that corrupted multi-line JSON for thinking-mode models, and a model took the blame for a protocol failure it did not commit. Under clean REST capture it passes all six probes.

That finding is Papers 1.16 and 1.19, and it stated the thesis of this page in a better sentence than the one at the top: The Model Did Not Fail The Protocol. The Terminal Did.

How to tell the difference

Two tests, both cheap, both used above.

When every seat agrees, inspect the mechanism before naming a defect. One consultant-bench task fails on the local 27B, the local 35B, a headless 2080, a Ryzen AI MAX 390, a hosted DeepSeek control at n=13, and a frontier model with reasoning enabled. The task is nevertheless solvable: three get_year_over_year_growth calls, one per company, fit inside its five-action budget, but require aggregation across category rows and company-level YoY arithmetic. Every tested model instead chose the intuitive six-call portfolio route and spent its budget before committing. That is a reproducible planning-budget failure mode, not a bench defect or a general ranking.

Does it survive a change of harness? A 2080 reproducing another card's exact failure fingerprint says the failure belongs to the model. The same model truncating mid-JSON on a different machine says it belongs to the harness.

Since this page was published

Twenty more, 12–28 September. Four upstream, sixteen mine — the same ratio, and the same property: each produced a number rather than an error. Two of them are the reason the cell model now has a third outcome, unproven, which is not a score and is never pooled.

What broke	What it looked like	What it was
`offloaded N/N layers to GPU`	A fully GPU-resident mixture-of-experts model	The counter counts dense layers. Ollama resolves an oversized MoE by moving every expert tensor to system RAM, keeps the dense weights on the device, and logs `31/31` anyway. A 17.33 GB model on an 8 GB card had never been resident and could not be — and the row was published as fully resident.
LiteLLM's `ollama_chat/` provider	A model that cannot use tools: it calls one, then calls it again, and loops	The provider drops tool results, so the agent never sees its own output. Routing as `openai/<model>` against `:PORT/v1` fixes it entirely.
`pkill -f` issued over ssh	A cleanup that reported success	The pattern matches the `pkill` command's own line, so it kills itself and leaves the target running. Resolve PIDs via `/proc/PID/environ` and signal those.
Pinning a card is not clearing it	A slower model	Residual occupancy. Same model, same task, same pin: 28.23 s on a card that was pinned but never verified empty, 19.75 s with the card checked down to its 54 MiB idle floor first.
My own placement probe	A 7x slowdown in a repeat run	The probe loaded a second model on the unpinned endpoint with a ten-minute keep-alive. The benchmark model then got 4,468 MiB of a 17,742 MiB blob. The run was voided and a residency-magnitude check added.
Turn loop recorded each turn after parsing it	“found 0 JSON objects” — a model emitting nothing usable	The evidence was discarded on the failure path, so the cell retained the previous turn's text. The turn is now recorded before parsing, and an empty response is a carrier fault rather than a model result.
Wall-clock trial count	A 2.1x speed gap between two models	n=3 on a task whose spread reaches 84% of its own mean. A paired n=6 run puts the same two models 1.6% apart with overlapping ranges. At three trials the draw was picking the winner, and would pick differently on another night.
`pgrep -c dispatch.py`	A healthy run reported dead	No `-f`, so the pattern never matched a script name sitting in an argument list. The same watcher also read MST timestamps as UTC and invented an hour-long gap.
A deterministic grader that had never been boundary-tested	17 failed cells across a matrix	False negatives. A grader untested against negation, synonyms and section headings is a suspect, not an oracle — which is why grader is now one of the five proofs a cell must satisfy.
Fixture path resolution on the Z13	Three different pass counts for one model: 12, 10, 9	The data client resolved `~/Downloads/SensorDownload/Current` ahead of the project's own data directory, so each run graded against whatever happened to be there.
Scope check counted the carrier's own files	A model capped at 6/18	Runtime pollution — `.pi-agent`, `.pi-sessions`, `.tmp` — written inside the fixture root by the harness. The model's edits were correct.
Scorer searched for literal scope markers	Six cells failing a scope check	It recorded no evidence at all, so a genuine pass and a miss were indistinguishable after the fact. v0.2 records which markers it searched for and what it matched.
`--placement-profile` set as an environment variable	Placement silently unproved	The runner takes it as a CLI flag. Set in the environment it was ignored, and every cell recorded `placement: false` while the run otherwise looked normal.
Unified-memory placement read a cold daemon	“no loaded model reported by the local daemon”, every cell unproven	`/api/ps` is empty until something is loaded, and the proof ran before the model did, so it never had a chance to pass. It now warms the model once and re-asks.
Dashboard row filtered by benchmark name	A completed run simply absent from the dashboard	Name-matching dropped a paired comparison that arrived under its own label — the same defect that had already dropped a whole generation of rows once and been fixed only in the one place it was noticed. The test meant to catch it asserted per definition, not per row, so a dropped row inside a populated definition passed it.
The carrier sandbox held the answer key	A model solving a task	`native_carrier.sh` bound the whole harness directory beside the fixture, so every task's postcondition regex sat one `ls` away. Largely unexploited, except gemma4 on ambiguous-anchor: 3/6 with the leak, 0/18 without.
The pin named five sampling parameters too few	Model-versus-model gaps	It fixed context, temperature and thinking, and left `top_k`, `top_p`, `min_p`, presence and repeat penalty at each model's default, so no two seats shared a sampler. One parameter alone moved qwen3.6 from 4/16 to 9/16 on one task. Every tag is now pinned on all fourteen options and checked before launch, and every request is checked in flight.
A repair fixture shipped already repaired	Six clean passes per model on the booking task	A 15 September edit left the corrected code in the fixture, so the task passed without a change. The runner now grades every repair fixture untouched and refuses to dispatch one that already passes.
`qwen3-next:latest` is the Thinking model	A weak seat: 10/30	The library's `latest` tag is the Thinking variant, which has no non-thinking mode. Run with thinking off in a 16k context, one trial produced 56,000 characters of reasoning against 2,700 of answer. The Instruct variant, fully pinned, scores 27/30.
The Vulkan guard, again	GPU1 running slower than GPU0	The first row on this page, reintroduced in a new script that set `CUDA_VISIBLE_DEVICES` but not `OLLAMA_VULKAN=0`, so each daemon also ran models on the other card. A per-cell placement gate caught it and the run was discarded.

Still not one of them was the model. Thirty-seven faults now. The closest thing to a model result in the whole set is the invention rate — how often a model answers instead of saying it does not know — and that only became visible once a benchmark stopped scoring “invented” and “abstained” identically.

What this page is not

It is not a claim that the tooling is bad. Most of these are reasonable behaviours being read for something they were never reporting, and more of them are mine than anyone else's.

It is not finished. It is a count, and the count goes up every week that measurement continues. A week without a new row would be the surprising outcome.

And it is not an argument against publishing results. It is an argument for publishing the instrument alongside them.