Local Lane

Metrics Power Rankings

The complete rows use the same five repair tasks and three prompt-guidance levels: Plan supplied (former L2), Files named (former L1), and Goal only (former L0). Each has six draws per task/level (90 cells). Task time is observed task wall-clock. Complete rows are sorted by their full 90-cell observed totals; Flint is appended last per the operator's editorial direction, with only its 73 recorded cells timed. Loaded decode and TTFT retain their probe provenance; Bolt and Luna use one warm-up plus one measured streamed request (n=1), not three repeatability draws.

Measurement readings are rounded to one significant figure; exact outcome and sample counts are retained. Complete rows cover 90 cells each and are ordered by their unrounded summed task times; Flint is appended last per operator direction. Median task time for complete rows is across all cells. Flint's ~75 min is the estimated elapsed duration for the full grid; its 48.91 min is only the sum of 73 recorded task timers. The run elapsed 60 minutes before timing out. The Flint median is 5 s after one-significant-figure rounding, but it is the Repair L2-only median (4.7 s before rounding), not an all-level median like the other rows; treat it as a level-specific diagnostic, not a comparable table statistic. Extrapolating the 17 missing cells from prior task/level draws and applying the observed run-overhead ratio projects about 75 minutes elapsed for the full grid; this is an estimate, not a completed measurement. For comparison, Gemma4 26B's displayed 60 min is also summed task time for 90 cells, not verified elapsed run duration. Flint's row is an exception: the run reached its 60-minute wall-clock limit and timed out at 73/90 cells (17 unrun: all trial 6 across levels, plus L2 trials 5×2). Per the protocol, incomplete results are not ranked. This row reflects the operator's editorial judgment to place Flint last among measured rows — dual-card placement still could not finish in an hour — not a protocol-compliant rank. Its denominators (23/25/25) reflect the timeout, not a different task mix. Loaded decode and TTFT were not measured under this profile. Existing Qwen/Prism serving metrics retain their original prompt and route contracts: C1 and Prism used one warm-up plus three measured 128-token calls, with prompt lengths varying by profile. Bolt's displayed 80 tok/s and 20 s TTFT are n=1 client-stream measurements on a loaded prompt with llama.cpp on one RTX 3090; its separate short-prompt probe is not substituted. Luna's 100 tok/s and 2 s are n=1 Pi-client streamed measurements on a loaded hosted request; TTFT includes Pi process/OAuth overhead and output rate is not provider server decode. These probes are useful diagnostics, not a perfectly controlled server-side ranking. Luna's six-draw task evidence combines evidence_luna_control_20261016T043229Z_retry3 with evidence_luna_control_20261016T084220Z_draws2to6_draw*. The run did not capture a before/after subscription quota meter; token events are not quota deltas, and no API-list-price estimate is a subscription charge. The older GPT-5.6 Luna results farther below are a different benchmark.

What this is

I benchmark the three chips I have:

• RTX 3090, 24 GB, power capped at 320 W. I own two and run them singly: every ranked measurement on this page is one model on one card. The pair is a capacity arrangement, not a faster machine. Moved after this page was written: they now sit in the test bench, which is where every measurement dated 21 Sep 2026 or later was taken. The historical hardware-ranking rows below were produced while they were in the desktop (i9-9900KF).
• RTX 2080, 8 GB, originally in a headless test bench (i3-9100F, 15 GB RAM, Debian 12); now in the controller desktop. Its rows below predate the swap.
• Ryzen AI MAX 390 with Radeon 8050S and 27 GB of unified memory, in a Flow Z13, running Vulkan rather than ROCm.

The latest results below come from retained run packets identified by name, with their sample sizes and configurations. Luna's hosted battery now has six draws per task/level, matching the local rows' 90-cell denominator; its provider and sampler remain distinct. Older measurements remain historical evidence, not extra trials to add to new scores. Corrections are stated rather than numbers quietly replaced.

The earlier measurement standard, as of 8 Sep. Models run as shipped. Only context length, temperature, thinking (off), the prompt and the probe length are pinned. Speed means measured task wall clock under a realistic prompt, not decode rate on an empty cache. That historical ranking describes each model at its shipped defaults, not at its best; the new matched cohorts declare their shared sampler separately.

The seat is qwen3.6:35b. I pick the seat from interactive use. The ranking informs that choice, it does not make it.

The descriptive guidance names apply to the current matched repair battery. Archived experiments retain their original L0/L1/L2 identifiers because older protocols may use different rung semantics; do not map those historical labels onto the guidance conditions above.

An earlier version of this page ranked the two chips by decode rate. It is archived in the site repository, and the sections below explain which of its numbers were wrong and why.

Native Fusion results (separate workload)

Fusion is a separate multi-step tool-use workload, not part of the repair table or its ranking. The median task times below are model- and profile-specific; the two serving profiles must not be pooled.

TennisAgent Fusion v2, formerly called Fusion L3, is a separate multi-step tool-use benchmark. Six tasks cover linked evidence, missing and invalid data, weather provenance, count reconciliation and contrasting evidence states. Each profile ran six trials per task on one RTX 3090 on Testbench. The task set is shared; the serving profiles are not.

Runner, tools, frozen databases, model server and telemetry all ran on Testbench. The six-task canary passed for each model and is excluded from the table. All 224 request-gate checks, including canaries, passed in the Prism cohort. Model loading, fixture checks and before/after placement verification are outside the reported task timer. ¹ PrismML CUDA 12.8 cohort; ² Ollama 0.32.12 C1 cohort. Fusion medians are reported within each cohort only.

Same measured quality means the same pass count under this frozen, bounded, partly lexical grader. It does not establish general quality equivalence. Shared settings were 16K context, 4,096-token output cap, temperature 0.7, top_p 0.8, top_k 20, seed 1234, and thinking/speculation off. Models ran sequentially on GPU0; conversation cache was erased between cells, with normal reuse within a task. Placement was attributed before and after each cell, not monitored continuously.

Evidence packets: fusion-native-llama-20260930 (Prism profile) and fusion-native-c1-20261001 (Ollama C1 profile), complete. Tasks and grader were unchanged. Required fixture states and counts passed; optional missing weather-column and unrelated Babolat-data warnings remain documented in the Prism packet.

Repair quality by prompt guidance, not one blended score

The fresh repair packet runs the same five tasks six times in each guidance condition: 30 cells per condition, 90 per model. Goal only (former L0) supplies the goal; Files named (former L1) supplies filenames; Plan supplied (former L2) includes a plan. These labels describe prompt support, not a difficulty scale: less guidance makes the task harder. The packet and raw records retain their original L0/L1/L2 identifiers. It uses Pi as the coding-agent carrier, unlike the native Fusion tool loop above.

Plan supplied alone would have hidden the difference. Both models completed all 30 plan-guided tasks. With less prompt support, Qwen passed more cells, and Bonsai's total time across all three conditions was longer despite its faster decode. These are observed counts, not a general quality ranking from six repeats per task.

Loaded decode is a separate probe: 12,811 actual prompt tokens, 128 generated tokens, temperature 0, one discarded warmup and three measured calls, reported as the median. Bonsai decoded about 63% faster here. It is not an agent-turn throughput measurement.

Evidence packet: standard-l012-20260930-r2, complete. Same 3090 GPU0 and PrismML prism-b10743-adfffbe CUDA 12.8 fork, shared 0.7/0.8/top_k 20 sampler, 16K context, 4,096 output cap and seed 1234. Independent cells start with erased conversation cache and warm weights; Pi compaction is off. All 180 scored cells have attributed before/after placement captures, zero qualified-unproven outcomes, and all 1,581 request verdicts, including qualification, passed. Verification overhead is excluded from task times. The Ollama C1 runs use the same 90-cell task battery but are a separate serving-profile cohort, summarized in the headline table above.

Historical hardware ranking

These are the earlier hardware rows, not the new matched model comparison. Speed is measured task wall clock on two E9 fixtures, the mean of the warm trials with the first of each run discarded, normalised to gemma4:26b on the RTX 3090 = 1.00. Configuration: num_ctx 16384, temperature 0.8, thinking off, ollama 0.32.12 except the MAX 390 at 0.32.13. The mixed-denominator quality column has been removed; current quality is reported by campaign above.

Every row is one card. This is a single-GPU ranking: one RTX 3090, one MAX 390, one RTX 2080, each running the model by itself. That is how I run them, and it is the only configuration these numbers describe.

Two models were removed rather than re-measured, because they cannot be measured here at all. gpt-oss:120b is a 60.9 GB blob and qwen3-next is 46.9 GB, against 24 GB of VRAM. Any figure either one ever produced on this machine was a multi-card or partly-host-RAM number wearing a single-card label. They are mentioned under what a second card is for and are not ranked. gpt-oss:120b's quality counts stand where they are cited. qwen3-next's are withdrawn: they were measured with the Thinking variant run in non-thinking mode (see the Instrument Log).

Eight is the whole list, not a cut from a longer one. Four models have wall clock in the corrected configuration on a single RTX 3090, three on the MAX 390 and one on the RTX 2080, which comes to eight. Several further models have quality data but no wall clock under the corrected protocol, so they cannot be placed here at all. The ranking grows when those are re-run, not when the cutoff is moved.

Rows 2 and 3 are joint second. Bootstrapping the RTX 3090 ranking over warm trials and pass counts puts qwen3.8:27b ahead of gemma4:26b in only 79.1% of draws, which is not separable. Every other adjacent pair on that chip is ordered at 98.7% or better, and qwen3.6:35b takes first in every draw.

The RTX 2080 row is the weakest evidence in the table. That fixture ran three trials rather than four, so its warm mean rests on n=2 where every other row has n=3, and its long task was still falling across those two trials, 218.6 s then 124.5 s. Its two per-task ratios spread from 0.23 to 0.31, wider than the gap between several adjacent rows above it. Read the row as placing the card in this neighbourhood, not as fixing it at 0.27, and note the direction: with the long task still coming down, 0.27 more likely understates the card than flatters it. It is also the only roster member that runs on the 8 GB card at a usable rate, so there is no second 2080 row to check it against. Note it does not fit that card: its experts run from host RAM, as the correction further down sets out.

Decode speed does not predict finishing

Five models, two E9 tasks, n=2 each, dual RTX 3090, 5 Sep.

The fastest decoder is not the fastest finisher. gemma4:26b decodes 2.9x faster than qwen3.8:27b and finishes later, because it writes four times the output tokens. That is why the standard moved from decode rate to task wall clock. One correction to this run: its wall clock included about 4.5 s per trial of ledger bookkeeping, since removed. The ranking above is re-measured without it.

Decode rate also depends on the prompt. The same probe under a ~12.8k-token prompt, the loaded condition that matches my actual use, shows every model losing throughput, but by very different amounts:

Because the gap is model-dependent there is no conversion factor between the two. A decode figure means nothing without the prompt length it was measured under.

Across machines

The rows above interleave three chips on purpose. What they show when read by machine rather than by rank:

Quality rank transfers between chips. Speed rank does not. Both the desktop and the MAX 390 put qwen3.6:35b and qwen3.8:27b at ceiling with gemma4:26b below. There is no single chip multiplier either. The MAX 390 penalty runs from 1.95x to 3.40x against the same model on the RTX 3090 and reverses between tasks, so any single "N times slower" figure for this chip is wrong. That is also why the MAX 390's fastest model on the short task is not its fastest overall: gemma4:26b reads the prompt quicker than the seat and then loses three times the ground on the long one.

Stated confound: the MAX 390 ran ollama 0.32.13 against the desktop's 0.32.12. That affects cross-chip magnitudes, not ordering within a chip. MAX 390 quality is 18 cells per model, shown as a rate. Its numbers also depend on power state, so write the power state down or throw the number out.

A headless test bench with an 8 GB RTX 2080 reproduces the desktop's correctness (15/15 against the desktop's 28/30 on the same tasks) at 3.8x the wall clock, 141.1 s against 37.1 s. And capping VRAM on the 3090 is a good stand-in for a smaller card: a simulated 8 GB envelope ran gemma4:26b at 31.55 tok/s against a real 2080's 31.02, 1.7% apart.

What the 8 GB card can hold decides everything else about it. Same probe, num_ctx 16384, ollama 0.32.12:

The dense model is the smaller file and runs 8.2x slower, because only 24% of its layers fit.

Correction, 12 Sep 2026. The gemma4:26b row was published as fully GPU-resident and it never was. That row previously read "31 of 31" with no other placement column, which reads as the whole model on the card. It is not. gemma4:26b is a 17.33 GB MoE blob whose full residency needs about 16.7 GB of VRAM; the card has 8,192 MiB with roughly 7.4 GB free. Ollama's own fitting pass resolves that shortfall by moving every expert tensor to system RAM and keeping only dense weights on the GPU, then logs offloaded 31/31 layers to GPU. Both statements are true and they describe different things: the layer counter counts layers, not experts.

This is checked, not inferred. Ten separate loads of this model on this card, from 20:19 on 11 Sep to 02:52 on 12 Sep, every one recording all MoE tensors moved to system memory with a shortfall of about 11.5 GB. Free VRAM ranged 6,478 to 7,413 MiB across those loads and never changed the outcome. This model has never been GPU-resident on this card and cannot be.

What that does and does not change. The 31.02 tok/s figure stands as measured, but it is a mixed-placement rate, not a resident one, and it is fragile in a way a resident rate would not be: the experts are memory-mapped from the model file, so decode depends on whether host RAM can cache them. Measured on the same card at the same placement, with the host under pressure, the same model returns 4.0 tok/s with about 580 major page faults per token and 38% iowait, an 8x swing with nothing on the GPU changing. Quality is untouched and is reported separately: 15/15 on this battery, 17/18 across the 11 Sep runs. Mixed placement changes how fast a row was produced, not whether it was right.

The instrument lesson. offloaded 31/31 layers to GPU is not sufficient evidence of MoE residency, and neither is ollama ps, which reported this model at 1.22 GB with a "25%/75% CPU/GPU" split while nvidia-smi showed the process holding 7,202 MiB. Only the load-time fitting decision, or a per-device VRAM measurement checked against the model's known full-residency requirement, establishes where an MoE model actually sits. The measurement standard has been changed accordingly, and no MoE residency claim on this site now rests on layer counts alone.

Still open: the Ryzen AI MAX 390 MoE rows. No fitting-pass evidence was captured in any run on that machine, so the same defect can be neither confirmed nor ruled out there. It is also an APU with unified memory, where the device-versus-host distinction is not the same question. Those rows are flagged pending a fresh load with the fit log captured.

The bench remains the functional-testing host, and anything about memory stays on the desktop.

Memory, context and placement

Pin num_ctx. gemma4:26b ships with a default context of 262,144 tokens. Unpinned, its cache takes it from 19.7 GB to 39.3 GB and splits it across both cards, and decode drops from 214.4 to 139.9 tok/s (n=6 interleaved, ranges do not overlap). The 35% loss was the oversized cache, not the split.

Splitting a model across two cards costs almost nothing when it fits on one. With context pinned, moving four models between one card and two changed decode by -4.7% to +9.3%, and under a realistic prompt by -8.0% to +6.7%. Decode walks the layers in order, so the two cards never work at once, and what crosses between them per token is tiny. What splitting does cost is about 1.4 GB of VRAM.

The exception is prefill. Two of four models read their prompt much faster on two cards, gemma4:31b by 40% and qwen3.6:35b by 19%. I do not know why. My first explanation, that they were short of headroom on one card, was tested the same evening and did not hold.

What a second card is for, and what it is not. It buys capacity: room for a model that does not fit, a longer context that stays on the GPU, or two models resident at once. For a model that already fits, at a short context, it does not buy decode speed: splitting cost 1.4 GB and moved decode between -4.7% and +9.3%, because decode walks the layers in order and the two cards never work at once. If you run one model that fits, at modest context, one good card is enough.

Where capacity turns into speed (measured 28-29 Sep 2026 on a headless test bench with two RTX 3090s, a different host from the rows above, so compare ratios, not absolute figures):

• A filled context. gemma4:31b with 118k tokens actually in context decoded at 17.7 tok/s on two cards and 1.2 on one, once the cache pushed layers to the CPU.
• A long prompt. Two cards read prompts 157% faster at 118k tokens. The gain holds even at 7k (41% faster), where the model still fits on one card.
• Two different models at once. One per card, neither slows the other measurably. Both on one card, they cannot batch together and must swap: 3 to 5.4 times slower, all of it reload.
• A model that does not fit. qwen3-next (50 GB) ran at 72.6 tok/s on two cards against 25.6 on one. nemotron-3.5-lightning (25.4 GB) is borderline: it runs on one card at about two-thirds of its two-card speed.

Where the second card earns its place. At a configured context of 131,072 with a ~12.8k-token prompt, qwen3.8:27b ran at 61.4 tok/s on two cards and 11.2 on one, about 5.5x apart. This is a configured-capacity sweep, not a prompt-length sweep, and placement on the one-card side was not captured, so how much of that comes from spilling to CPU is provisional. Two models also fit side by side on the pair up to a context of 32,768, which makes switching between them take 0.6 s instead of a full reload.

Spilling to the CPU is expensive. Forcing layers of gemma4:26b onto the CPU (empty-cache probe, n=3) took it from 224.8 tok/s at 31 of 31 layers on the GPU to 172.7 at 30 of 31. One layer cost 23%. All four measured points fit a fixed 1.34 ms per CPU-resident layer. Whether that cost carries over to other models is not established. A rough estimate for qwen3.8:27b comes out near 5.5 ms per layer, but it divides a VRAM shortfall by an assumed layer count rather than counting layers, so it is arithmetic, not a measurement. And a dense model near its memory limit does not slow down smoothly: gemma4:31b at a reduced GPU layer count passed two tasks 3/3 and timed out on others. The limit is a stall with task-dependent odds, not a wall.

The interactive version of this is Where the Layers Go.

Archived E9 ladder: the quality ceiling and the noise floor

Small samples lie. One cell, gemma4:26b on csv-summarize-repair, had produced 5/6, 6/6 and 6/6 in separate runs, two of them read as clean results. At n=100 its true pass rate is 75/100. Six trials cannot distinguish that from 100%. Every result is now labelled with the evidence tier its sample size supports.

In this archived E9 ladder, the hard rung was saturated while the easier rung still separated models. These legacy L0/L1/L2 rungs are from an older protocol, not the prompt-guidance conditions in the current table. At legacy L2 the roster tops out the E9 tasks. At legacy L1 the same tasks still separate models: gemma4:31b and gpt-oss:120b 30/30, qwen3.8:27b 26/30, gemma4:26b 24/30. Three fixtures that had been written and never run came back 18/18 for all seven models tested. The top of the ladder is exhausted.

Decode class does not predict correctness either. nemotron-3-nano decodes at 121.2 tok/s and passed 0 of 18 cells, missing the implementation rather than timing out.

A tool-grounded consultant bench

The newest bench asks consultant questions about a real medical-device registry, 2008 to 2025, three companies. The model gets a catalog of registered tools and one question at a time, has to actually call the tools against the live database within three turns, then commit an answer as JSON.

Each answer is scored three separate ways: protocol (well-formed turns), execution (it really called the tools when the task needs data) and truth (the answer matches ground truth computed independently by SQL over a pinned copy of the database, never taken from the tool path being judged). One task is planted: it asks about 2030, outside the data, and the only correct answer is that there is none.

Overnight on 10 Sep, one RTX 3090 per model:

It ran on off-peak electricity for under a dollar, estimated from the utility rate rather than metered.

Hosted models on the same bench, as controls:

Task 05 fails on every model tested, local and hosted, including a frontier model with reasoning on. That uniformity turned out to name a mechanism, not a bench defect. CPB allows five tool actions. The intuitive route asks for each company's 2024 and 2025 portfolio separately, needs six calls, and exhausts the budget before a final answer. A valid three-call route uses get_year_over_year_growth once per company, then aggregates the returned category rows and computes company-level YoY. The repeated miss is therefore a tool-path, planning-budget, and arithmetic failure mode. It is not a general ranking.

Everything except DeepSeek is a single lap and cannot be ranked against hundreds of local laps. Within this harness, though, the local 27b passes tasks 01 and 03 on every one of 302 laps, where DeepSeek managed 3 of 13 and 0 of 13.

Testbench remeasurement, 24 Sep: a fresh native 30-lap same-seat replication at the historical 65K configuration and a separate clean 30-lap E9-pin leg both passed tasks 01, 02, 03, 04 and 06 on every lap and missed only Task 05. That is a configuration-invariance result within this bench: the same task pattern held across a 4x context change and temperature 1 versus 0.8. The two 30-lap legs are a configuration comparison on one RTX 3090, not a replacement for the historical high-n series or a general model ranking.

Two hosted controls are left out on purpose. Claude Sonnet and Claude Haiku were driven through a command-line path that still exposed my account's own connected tools. Haiku replied that the bench's tools were not in its toolkit and named its email tools instead, and Sonnet answered "no data" without calling anything. That is the harness, not the models. It also shows a property of the bench: a model that refuses the tools still passes the 2030 task, so 1 of 6 is the score for doing nothing.

What reasoning effort costs

The same six tasks on GPT-5.6 Luna through a subscription, one lap with reasoning off and one at medium, 10 Sep. Token counts come from the API's own usage report and are priced at list, $1.25 per million input tokens and $10 per million output.

Reasoning tokens are 72% of the output and about half the cost of the lap. Turning it on cost 4.5x as much and passed 2.5x as many tasks, which works out to 1.8x the cost per task solved. At list price, one lap is about two cents with reasoning off and ten with it on. A single lap did not visibly move the subscription's usage meter.

This is also what makes subscription value measurable: the bill is flat, and the list value of the tokens used is now counted per lap.

What this does not say

• It does not rank hosted models. Those rows are single laps, except DeepSeek at 13.
• It does not claim local models match frontier models in general. The consultant bench is six bounded tasks over one database.
• It does not explain the two-card prefill advantage. That is an open finding.
• It does not show any model at its best. Speculative-decoding depth is inherited from each model's defaults and differs across the roster. qwen3.8:27b runs at a draft depth of 4 against a measured optimum of 8 on the 3090, so its speed figures are a floor.
• It does not treat these numbers as fault-free. Seventeen measurement faults were caught while producing them, each one a number that looked like a finding. They are listed in the Instrument Log.

Correction and re-measurement, 22 Sep 2026

Every E9 row on this page was re-run on 22 Sep with a harness that records what it was measuring. 130 cells across both hosts, none of them unproven. The protocol those runs followed is now published separately, at Benchmark definitions, and generated from the same file the harness reads rather than restated here.

The hardware moved. The two 3090s are now in the test bench and the 2080 is in the controller desktop. The bullets above are marked accordingly. No row below was re-normalised for the swap, because a row means the machine it ran on.

Two claims above no longer hold.

• The MAX 390 ran qwen3.8:27b's long task in 48.0 s, not 96.8 s. That breaks the joint sixth place: qwen3.8 now separates from gemma4:26b on that chip rather than tying it.
• “The MAX 390's fastest model on the short task is gemma4:26b” is no longer true: the seat reads the prompt quicker, 20.2 s against 22.0 s.

The stated MAX 390 penalty of 1.95x to 3.40x also widens to roughly 1.5x to 4.0x against the same models on a 3090. The point that no single multiplier exists for this chip stands; the published range does not.

Placement is now proved per cell, and it was not before. The September native rows ran with the residency gate disabled and an empty process list captured at preflight, so where those models sat was never established. Runs now attribute GPU memory to the measuring daemon's own process tree and fail closed if a model spans cards. The 22 Sep rows were taken on one pinned card.

Pinning a card is not the same as clearing it. The 22 Sep rows ran on one pinned card that was not verified empty first, and a partly occupied card is slower in a way nothing in the trace records. Runs now check the card down to its idle floor before launch. The effect is not uniform: re-measured on a cleared card, qwen3.6:35b's long task came in at 19.8 s against 28.2 s, while qwen3.8:27b's held at 20.1 s against 18.8 s.

The evidence that legacy L1 still separates models is thinner than it reads. Above, the legacy L1 result is given as gemma4:31b and gpt-oss:120b at 30/30, qwen3.8:27b at 26/30 and gemma4:26b at 24/30. Both models at the top of that list are now off the single-card roster: gpt-oss:120b at 60.9 GB cannot produce a single-card number at all, and gemma4:31b was ruled off on 26 Sep — at Ollama's default 32k context it puts 7% on the CPU and takes 74–213 s per call, and at the pinned 16k it runs in about 10 s but is still not an E9 model. So the spread that makes L1 look discriminating is carried largely by models that are not in the race. Whether legacy L1 separates the eligible roster is an open question, not a finding — and it matters, because legacy L2 is saturated and legacy L1 is where the ordering would have to come from.

The long-task gap between the two qwen models is not established. The ranking above has them 2.1x apart on the long task on a 3090, 15.6 s against 32.4 s. A paired run on 25 Sep — single 3090, same daemon, minutes apart, cleared card, num_ctx 16,384 and temperature 0.8 on both, six warm trials each — put them 1.6% apart: 19.8 s against 20.1 s, with ranges that overlap almost entirely. The short task is not in question and moved the other way, widening to 2x with non-overlapping ranges, 7.6 s against 15.1 s. That run is on the test bench's 3090 rather than the desktop's, so it does not re-normalise the row; what it removes is the finding. The long task's spread reaches 84% of its own mean between identical trials, so at n=3 it cannot resolve a difference this size and the draw decides the ordering. The seat's first place rests on the short task and on quality, both of which hold; its 1.99 speed figure does not.

Some failures were never capability failures. Two kinds were being summed into one pass count:

• Out-of-scope files. gemma4:26b writes scratch test files (test_tmp.py, test_solution.py, repro.py) while solving a task, and the scope check counts that as a failure. Most of its published legacy L2 gap is this, not a wrong answer.
• Context truncation. Nine failed cells across both hosts ended at the pinned 16,384-token ceiling with no answer to grade — the battery then reported an empty result, which reads exactly like a wrong implementation. Part of gemma4:26b's collapse at the easier legacy rungs is this.

Both are now recorded as their own classes. A pass count that mixes them with capability misses is measuring three things and reporting one.

And one instrument fault, caught live. Loading gemma4:26b on 22 Sep, ollama ps reported 1.17 GB while nvidia-smi showed 19,282 MiB held by that process. The Instrument Log already lists this API as unreliable for footprint; it remains unreliable, and no placement claim here rests on it.

Related

• Where the Layers Go: interactive layer spill and the decode-to-wall-clock inversion.
• Qwen 3.6 vs 3.8: same architecture, new weights, and the ceiling item the local field sits under.
• Benchmark definitions: what each benchmark measures, generated from the harness's own definition file.
• The Instrument Log: every measurement fault caught in this program.
• Capture Integrity: the model did not fail the protocol, the terminal did.
• Papers: The Default Draft Is Too Deep, Dedicated 24 GB Beats Unified 27 GB, Slow Is Not Smart, When Local LLM Clusters Do Not Help, Local Models Cost Frontier Tokens.