benchmark_definitions.yaml
What each benchmark measures, how many cells it takes, what voids a run, and what may never be pooled. The numbers live elsewhere; this page is the protocol behind them.
A result is only as reproducible as the stack that produced it. Where a value was not recorded, this page says so rather than leaving it out.
A benchmark result without its protocol cannot be checked, and a protocol restated in prose drifts away from the runs it describes. Every definition here is generated from the file the harness and the internal dashboard both read, so there is one source rather than three descriptions that agree until they do not.
| Branch | Question | Examples | Interpretation boundary |
|---|---|---|---|
| Task capability — E9 / L0–L2 | How does task correctness change as instructions become less explicit? | E9 standard L2 and its L1/L0 extensions | Related instruction levels, not one pooled score; report each level and host with its own denominator. |
| Multi-step tool use — TennisAgent Fusion | Can the agent complete linked, multi-step tool workflows against a bounded fixture? | Fusion v1 retained; Fusion v2 current | A separate task family and grader, never an E9 rung. |
| Hardware and runtime performance | What do fit, placement, context, prefill, throughput, and service conditions do to measured performance? | Placement crossover; context/prefill studies; serving probes; dual-card study | Descriptive performance lanes; runtime, artifact, prompt, and memory regime must stay attached. |
| Interactive responsiveness | How quickly does a service return text for ordinary short prompts? | Five matched short prompts across TestBench, Z13, and hosted Luna | Client-observed service latency, not answer quality or a hardware-only ranking. |
Staged controls and applied-domain validation Staged frontier controls and real-data domain validations test a workflow boundary or application path. They are not another pooled model-ranking branch. The Tennis linked-data Luna observation is a domain-specific n=3 case, not E9, Fusion, or a service-latency run.
| Definition | Status | What it is |
|---|---|---|
e9-standard-l2 | published standard | The published local-lane E9 standard. Two linked legs over different task sets: a bounded L2 quality gate and a wall-clock battery. Wall clock ranks; quality gates. |
e9-extension-l1 | extension | The same two legs at L1. Run because the roster tops out the L2 quality gate. An extension, not the published standard. |
e9-extension-l0 | extension | The same two legs at L0, the level that currently separates the roster. An extension, not the published standard. |
fusion-v1-retained | retained not reproducible | Six multi-step tool-use task classes over the TennisAgent fixture, three models, n=3 = 54 cells. A separate benchmark family from E9: not an E9 instruction level, not a Haiku adapter, and never pooled with either. |
fusion-z13-diagnostic | diagnostic only | A Z13 run against the Fusion L3 task lineage under a changed wrapper. Preserved as raw evidence; it is not a protocol anyone should re-run or score. |
fusion-v2 | current | A deliberately new, separately versioned packet built after the retained one could not be recovered: one six-task manifest, one native runner, one deterministic grader, model-free boundary tests and a native-host contract. |
staged-frontier-control | separate control | The E9 task lineage run through a staged carrier -- text planner, deterministic fixture executor, synthesis, deterministic grader -- as a frontier control. Not a fourth local model and not a GPU speed measurement. |
blind-read-eval | single observations | Anonymised delegate-brief responses judged by a sandboxed Claude, with the operator's own ranking recorded separately. Four rounds, one attempt each. |
dual-card-placement | current | What a second card does and does not buy. The model is held roughly fixed and the card configuration varies, which is the inverse of every model benchmark here. Requires two cards by construction and can never appear in the single-card ranking. |
ppr-consultant-bench | current unverified under new harness | Tau-bench-shaped and execution-grounded. The seat gets a tool catalog and one consultant question per turn, and must execute registered tools against a live medical-device registry (2008-2025, three companies) rather than answer from memory. At most three tool turns, then it must commit a JSON answer. |
ppr-consultant-bench-card0 | current | The same protocol as CPB on a different single-card seat, built to be cross-seat comparable: CPB ran a 35b depth65536-qwen38 on card1, C0B a fresh qwen3.8:27b on card0. Self-contained; never writes into the CPB tree. |
hybrid-determinism | current | Answers whether a model is faster and more correct alone (raw) or answering from a BT domain's deterministic result (grounded). Built for the "is deterministic AI faster" question: pure deterministic AI is not testable, so the test is of hybrids. 247 questions generated from ppr_agent.db over six templates, one try per mode per question, graded correct / abstained / wrong against the SQL result. |
ppr-anchor-integrity | built never run | Built to fill a gap the other PPR benchmarks leave: each of those tests one seat, and this one tests whether an answer stays faithful to its anchor. |
tennis-linked-data-staged-validation | single observations | One staged TennisAgent planner → local deterministic Watch/Zepp tools → hosted GPT-6 Luna synthesis run over a real, read-only session fixture. Three valid-data and three invalid-date trials. Strict reconstructed checks scored 5/6; a documented semantic review scored 6/6. This is a domain-specific integration observation, not an E9, Fusion, model ranking, or performance result. |
alignerr-usecase | retired saturated | Three tasks derived from saved third-party work artifacts, scored per lane. Carried as a possible replacement instrument after E9 saturated. |
Every ranked measurement is a single GPU running a single model. Card count is an identity-critical component of a result: a row measured on a different number of cards is not comparable, by definition rather than by convention.
| Host | Label | Hardware | Contract |
|---|---|---|---|
z13-max390 | Z13 local GPU | Flow Z13, Ryzen AI MAX 390, Radeon 8050S, 27 GB unified memory | One integrated GPU with unified memory. There is no second device to split across and no device-versus-host VRAM question to ask, which is why placement here proves endpoint identity instead of residency. Vulkan (not ROCm) |
testbench-native | Testbench native | RTX 3090, GPU 0 (GPU 1 carries the interactive 32k route and is not benchmarked) | One. Two 3090s are on hand and they are run singly; the pair is a capacity arrangement, not a faster machine. Splitting a model that already fits cost 1.4 GB and moved decode between -4.7% and +9.3%, which is noise: decode walks the layers in order, so the two cards never work at once. runner, fixture, Pi, Ollama endpoint and placement telemetry all on Testbench |
desktop-single-card | Desktop, one RTX 3090 | RTX 3090, one card, i9-9900KF, power capped 320 W | One. The PPR benches pinned a seat to a single card and a single daemon port (CPB card1 :11436, C0B card0 :11435) so the two are cross-seat comparable. |
desktop-dual-3090 | Desktop, both RTX 3090s | two RTX 3090, 49,152 MiB pool, i9-9900KF | Two, deliberately. This host exists only for the hardware study, where the card configuration is the variable. No model benchmark may use it. |
testbench-hybrid | Testbench hybrid (deprecated) | diagnostic_only | Model inference on remote Testbench via the l3-testbench SSH tunnel with the fixture and tool runner on the controller desktop. Routing was correct; the old placement gate wrongly inspected the controller's local nvidia-smi. Preserved as hybrid diagnostic evidence. Never presentable as a Testbench-native row. |
A model that does not fit on one card is out of scope, not slow. gpt-oss:120b (60.9 GB) and qwen3-next (46.9 GB) against 24 GB cannot produce a single-card number at all, and any figure they carried was multi-card or partly host RAM under a single-card label. Removed from the ranking rather than re-measured. They are named here so the omission is visible, and are not ranked.
The current screening specification is recorded separately from the benchmark
results and is marked spec_only_never_run as a unified campaign. That
does not mean there are no historical screening observations: prior L2-only roster
screening and later prompt-level studies exist, but they used different runs and
harness contexts and must not be presented as one exact replay. The dedicated
screening page carries the roster evidence and its limits.
Placement on older rows. Results from before 22 Sep 2026 ran without a placement proof, so for those rows it cannot now be established whether the model was on one card, two, or partly in host RAM. New rows carry the evidence; old ones carry the gap.
They are instruction levels of the same task against the same postcondition. Only the prompt's specificity changes: never the fixture, never the grading. A level measures how much the model has to infer, so a model that passes L2 and fails L0 has not become less capable — it has been told less.
| Level | Prompt is | What the model must do | Example prompt |
|---|---|---|---|
L2 | fully specified | Numbered steps naming the file, the exact strings and the verification. The model executes rather than infers. | "1. Read runbook.md... 2. Under the heading '## Staging' only, replace the single line '2. Run: restart the service' with '2. Run: restart the service --force'..." |
L1 | stated plainly | One clear paragraph naming the file and the intent, without step-by-step or exact literals. The model decides how. | "In runbook.md, the step 'Run: restart the service' appears under Production, Staging and Development. Change only the Staging one..." |
L0 | oblique | How a colleague would actually ask. No file named, no literal given, the goal implied. The model must infer the target, the exact edit and the success condition. | "Staging needs the force flag when we bounce it. The other environments must stay as they are." |
The postcondition does not soften as the prompt does. At L0 the grader still requires the exact byte match it requires at L2, so an under-specified prompt is graded against a fully specified answer. That is the intended difficulty and also its main confound: a level measures instruction inference AND exact-form luck together.
On the Z13, gemma4:26b scores 6/6 on ambiguous-anchor at both L2 and L1 and 0/6 at L0, with no modified line appearing in any trace and half of those failures ending in context truncation. It does not do the edit at all rather than doing it wrongly.
There is no L3, and there never was. The name used to appear as
“Fusion L3”, where L3 meant a third difficulty tier on the Fusion axis
— a separate benchmark family with its own tasks, fixture and grader. Sitting
next to L0/L1/L2 it read as a rung above L2, which it never was. That family is now
TennisAgent Fusion, versioned v1 and v2, since versions are what
actually distinguish the packets. Its artifact paths still contain
l3 and are deliberately unchanged, because every evidence root and source
link on file points at them.
The quality leg and the wall-clock leg run over different task sets, in separate runs, to separate evidence roots. They share exactly one task. A quality count therefore does not vouch for a time recorded beside it, and warm means include failed trials unless a row says otherwise.
| Quality leg | Wall-clock leg | |
|---|---|---|
| Measures | correctness | end-to-end task wall clock |
| Tasks | ambiguous-anchor, csv-summarize-repair, strict-log-format | constant-and-callers, csv-summarize-repair |
| Trials per task | 6 | 4 (discard cold t1) |
| Reports | pass count over 18 cells | warm t2-t4 mean, plus individual warm trials |
| Role | bounded gate and noise floor — NOT the model differentiator | the differentiator, once the quality gate is cleared |
| Read it with |
|
|
Pinned across every E9 definition:
tag=e9pin, num_ctx=16384, temperature=0.8, thinking=off, ledger=none.
Everything else runs as the model ships, so a ranking here ranks each model at its
shipped defaults, not at its best.
Pass and fail are not enough, because a harness fault and a wrong answer look identical in a pass count. Both produce a number rather than an error.
A failing cell is promoted to fail only when all five proofs hold:
| Proof | What it establishes |
|---|---|
dispatch | the call happened — non-zero tokens, a resolved model identity, no no-dispatch state |
scope | the graded fixture is byte-identical apart from the model's own intended edits |
placement | the model ran where the row says it ran, from the host's own telemetry |
grader | the failure survives a grader spot-check. A deterministic grader that has never been boundary-tested against negation, synonyms and section headings is a suspect, not an oracle. |
timing | the trial completed without timeout, and the recorded time excludes harness bookkeeping |
Default to unproven. Promoting unproven to fail requires the proofs, not an argument that the model probably was wrong. Mixed results across a roster are the signature of an unproven cell class, so triage the instrument before interpreting the spread.
| Class | Meaning |
|---|---|
exact | All identity-critical components match and no blocking-until-bridged component differs. A recorded patch-level delta does not disqualify a row. |
constrained | Identity-critical components match, but a blocking-until-bridged component differs, or the operator recorded the run as constrained at the time. A valid measurement of the model; not by itself proof that the historical row reproduces. The September z13 L2 rows are constrained because their closeout says so — that label is the operator's, and a bridge run is what would revisit it, not a re-reading of the version numbers. |
original | A first measurement with no historical counterpart — the L0/L1 extensions, and every native Testbench row. |
One harness across three point releases, run weeks apart: historical 0.85.1, testbench 0.86.0, z13 0.86.1. Drift is normal and cannot be prevented — work that spans months will span releases. Recording it is the requirement; freezing it is not.
| Bucket | Rule | Components |
|---|---|---|
| identity critical | must match, or rows are not comparable at all | task hashes, rendered fixture hash, model tag and digest, sampling options, leg structure, host class, card count |
| recorded not blocking | Stamped on every row so a question can be asked later, but does not by itself demote a row or require a re-run. A patch-level harness or server bump lands here until there is evidence it moved a number. | harness patch version, ollama patch version, runner revisions that do not touch measurement |
| blocking until bridged | Known to change what is measured, so rows either side are not pooled until a bridge run says otherwise. | runner changes that alter the measurement (2026-09-07 ledger removal, 09-11 residency gate and endpoint pin, 09-16 host gate), harness major or minor bumps |
Bridge run. The cheap way to settle a suspected-material change: re-run one cell class on both versions on ONE host, small n, and compare. A bridge run answers the question for the whole corpus. Absent one, do not assume a version delta explains a result — and do not assume it does not.
Pooling is forbidden by default: results may only be compared within one definition, on one host.
Counts accumulate over different corpora with different denominators. A rate computed across them is arithmetic, not a measurement. Quality rank transfers between chips; speed rank does not. The Z13 penalty against the 3090 varies by task and reverses, so no single multiplier exists.