Every measurement fault caught in this program, and the finding it would have been published as.
The harness doesn't just decide outcomes. It decides what you're able to believe about outcomes.
Fourteen faults, caught over roughly a week of local-model benchmarking on two RTX 3090s. Not one of them was a model behaving badly. Every one was the instrument, the daemon, the log parser, or my own code.
They are collected here because of a property they share: each one produced a number rather than an error. A crashed run announces itself. A daemon quietly using two cards while the row says "solo" does not. It produces a clean, plausible, publishable figure that happens to be false, and nothing downstream can tell the difference.
That is the argument for the log. Anyone can report what their system did. The question worth asking is how many of these were sitting inside the run that produced the report.
Five faults in things I did not write. These are the expensive ones, because there is no commit to review.
| What broke | What it looked like | What it was |
|---|---|---|
CUDA_VISIBLE_DEVICES does not hide a card from Vulkan | A valid single-card measurement | Two cards. The daemon enumerated Vulkan0 as the other GPU and allocated on it. One config died on "failed to allocate Vulkan0 buffer". Requires OLLAMA_LLM_LIBRARY=cuda_v13 as well. |
utilization.gpu | "One card computes while the other idles, so layer-split is a pipeline" | The idle card was drawing 234 W against a measured idle baseline of 26-37 W, and climbing 70 to 78 degrees C while its reported utilization fell from 5% to 1%. Both cards were working. Power is the signal that holds. |
pcie.link.gen | Flaky hardware: a slot that "flipped gen1 x16 to gen3 x16 in one day" | Link power management. The gen downshifts at idle and negotiates up under load, so it means nothing unless sampled during generation. Width is topology and is stable. |
ollama ps | Model footprint, placement and context depth | Reported 29 GB against 34,728 MiB actually allocated. Reported "100% GPU" for a runner with a layer explicitly held back on CPU. Reported a 262,144 context that was not allocated at that depth. |
load_duration | Cold-load cost | 0.001 s on one daemon and 0.52 s on another, for the same already-resident model. A reporting difference between versions, not a load. |
Nine faults in code I wrote or maintain. More than the upstream count, which is the honest ratio and the reason this page exists.
| What broke | What it looked like | What it was |
|---|---|---|
Layer capture parsed journalctl -u ollama | status: ok, no placement concern | A hand-started daemon is not under that systemd unit, so its placement was never visible. A configuration with a fifth of the model on CPU recorded as a clean pass. Only a VRAM shortfall and a 5x throughput drop gave it away. |
Eviction check used bare ollama | Checking the daemon under test | Always talked to :11434 regardless of which daemon was being measured, so a solo run's eviction check inspected the wrong process entirely. |
| Dashboard glob card | One card failing to render | An unguarded relative_to on a path outside the repo took down the entire dashboard render, not just that card, in a generator whose own documentation says cards must degrade to an explicit empty state. |
Hardware snapshot captured power_limit | Provenance | A static 320.0 on every row forever. The field that turned out to be the reliable activity signal, power.draw, was not captured at all. |
| Ledger writes inside the wall-clock timer | A 25% difference between daemon versions, 11-12 s against 15-16 s | Bookkeeping overhead. Two correction sections were written attributing it to the ollama version before anyone opened the traces. |
prompt_s_median stored to one decimal | "Prefill is dead flat at 0.872 ms/token across three context depths" | Flat within the recorded precision. All three depths logged 11.2 s, so the identical figures were partly a rounding artifact. The conclusion survived. The phrasing did not. |
| Bench task authored out of scope | A model failing a task | The task reached outside its own fixture, so the failure was the bench. |
| Token usage discarded on every turn | "This endpoint does not return usage, so subscription spend is unmeasurable" | The endpoint returns it richly, including reasoning tokens and a per-message attribution breakdown. The client parsed it correctly and the caller bound it to a variable it never read again. Every lap since the file was written was unpriceable for that reason alone. |
| State-block generator was not idempotent | A working tool | The document gained a newline on every regeneration. Caught by the tool's own --check mode on its first run, which is the only reason it is a line in this table instead of a slow corruption. |
Five consecutive correction sections were written into a working handoff about a single wall-clock discrepancy. Four of the five were wrong in whole or in part. The only correct one was the only one that opened a fixture before writing.
The first asserted an attribution from reasoning. The second inherited that premise without checking and turned it into a standing instruction about which figures to cite. The third read the traces and found the arms inverted. The fourth confirmed the third and caught two further defects. A fifth then retracted the third's mechanism as well.
The measurements were never the problem. The fixtures were on disk the entire time. What failed was documents asserting things about artifacts nobody had reopened, and each one made the next reader more confident. "Correction #3 says X" is not evidence. It is a claim with exactly the standing of the one being corrected.
This log starts in September because that is when I started counting. The oldest entry is older and was written up properly at the time: subprocess capture embedded VT100 terminal sequences that corrupted multi-line JSON for thinking-mode models, and a model took the blame for a protocol failure it did not commit. Under clean REST capture it passes all six probes.
That finding is Papers 1.16 and 1.19, and it stated the thesis of this page in a better sentence than the one at the top: The Model Did Not Fail The Protocol. The Terminal Did.
Two tests, both cheap, both used above.
When every seat agrees, suspect the harness. One task in the consultant bench fails on the local 27B, the local 35B, a headless 2080, a Ryzen AI MAX 390, a hosted DeepSeek control at n=13, and a frontier model with reasoning enabled. Seven seats, one failure. That is a bench defect, not a ranking.
Does it survive a change of harness? A 2080 reproducing another card's exact failure fingerprint says the failure belongs to the model. The same model truncating mid-JSON on a different machine says it belongs to the harness.
It is not a claim that the tooling is bad. Most of these are reasonable behaviours being read for something they were never reporting, and more of them are mine than anyone else's.
It is not finished. It is a count, and the count goes up every week that measurement continues. A week without a new row would be the surprising outcome.
And it is not an argument against publishing results. It is an argument for publishing the instrument alongside them.