How models get ruled out

Eligibility comes before quality

A model that cannot hold itself on one card is out of scope, not bottom of the table. That distinction is the whole first stage, and getting it wrong is how multi-card and host-RAM figures ended up published under a single-card label.

Four checks, all of which must pass before a single quality cell is graded:

• The whole model resident on one 24 GB card at the pinned context.
• Resident share at least 0.9 of the blob size the daemon reports.
• Placement proved from host telemetry, attributed to the measuring daemon's own process tree — not from ollama ps, which reports dense layers and will happily say 100% for a mixture-of-experts model whose experts are in system RAM.
• The card cleared to its idle floor before load. Pinning a card is not clearing it.

Ruled out, and on what grounds

The gemma4:31b row is the useful one. It is a capable model that is simply outside the operating envelope of this configuration, and a quality score for it would really have been a placement score.

The ladder, and where it ran out

Three instruction levels over the same tasks, fixtures and postconditions. Only the prompt's specificity changes, and the grader never softens: at L0 the exact byte match required at L2 is still required. The numbers run opposite to difficulty — L2 gives the most help.

L2 was known to be exhausted. L1 was the rung the screening design picked to order the roster, and the campaign measured it directly:

87 cells, one card cleared to 54 MiB and evicted between models, placement proved per cell, zero unproven. Every eligible model is at or within one cell of the ceiling, so L1 joins L2: closed for ranking, kept as a gate, ordering moves down a rung.

One earlier claim dies here. "L1 still separates models" was published on the strength of gemma4:31b and gpt-oss:120b at 30/30 against two lower scores. Both of those models are now off the single-card roster, so the spread that made L1 look discriminating was carried mostly by models that are not in the race. On the models that remain, it separates nothing.

Where they actually differ

L0 was run three times on a repaired harness — six trials, then twelve, then sixteen — for 34 cells per model per task. Read summed, it says almost nothing. Read per task, it says a great deal.

Three shapes, not three positions. gemma4:26b is simultaneously the worst model in this battery and the best one: it never once passes ambiguous-anchor across thirty-four attempts, while beating the seat on strict-log-format. Which model “wins” depends entirely on which task you ask, so a single ordering throws the finding away.

The per-task differences were significant long before the summed one was. gemma4:26b has now failed ambiguous-anchor thirty-four times out of thirty-four — and on strict-log-format it beats qwen3.6:35b, 31/34 against 22/34. The same model is last overall and second on that task. A single ordering hides that completely, which is why the summed column is not the one to read.

The test that failed, then passed

Two runs at six trials had qwen3.8:27b ahead of the seat by four to five cells at L0 — enough to look like an inversion, since the seat ties at L1 and wins every speed comparison. So it was tested properly: twelve trials, with the decision rule written into the run directory before the run started.

The rule was a two-sided Fisher test on that run alone, p < 0.05, deliberately not pooled with the earlier runs. The result was 31/36 against 23/36, p = 0.055.

That fails. So on that evidence the L0 advantage was not established.

Disclosed because leaving it out would be the dishonest version: a one-sided test on the same data gives p = 0.027, and the direction genuinely was predicted by two prior runs. But two-sided is what was registered, and choosing the test after seeing the numbers is the thing pre-registration exists to stop. The lesson is for the next one: register one-sided when the direction is predicted in advance, not afterwards.

The fourth run

It was run once more at sixteen trials — 48 cells per model — with the rule again fixed in advance, and this time registered one-sided, because the direction had been predicted by three prior runs and the lesson from the previous failure was to say so beforehand rather than afterwards.

qwen3.8:27b 42/48 against qwen3.6:35b 32/48. One-sided p = 0.014 — established. It also clears two-sided at p = 0.027, so the result does not depend on the choice that was registered.

Four runs, two harnesses, three points of drift, and it took about 150 cells per model to establish something visible in the first eighteen. The six-trial result pointed the right way and could not carry the claim. That is the cost of a denominator chosen by convention rather than by the size of the difference being tested.

What this does and does not mean. The seat, qwen3.6:35b, ties at L1, wins every speed comparison, and loses this one. That is a difference between two models at one rung — not a roster-wide inversion, and not a statement about architectures, since there is one model of each here.

The control, which was also pre-registered, is significant. gemma4:26b is the same size class as qwen3.8:27b. If the effect were about parameter count, the two should sit together. Instead gemma4 came last. At twelve trials it was worse than qwen3.8 (p = 0.002) and indistinguishable from the 35b (p = 0.34); at sixteen it separates from the 35b as well (21/48 against 32/48, p = 0.040), so the earlier phrasing — that it “tracks the 35b rather than its size-mate” — is withdrawn. It tracks nothing; it is simply last. The claim that survives is the one that mattered: gemma4 is worst despite sharing qwen3.8’s size class, so the ordering is not parameter count.

Worth noticing what happened there. A null result was doing load-bearing work — “indistinguishable from the 35b” — and more data dissolved it. Absence of a detectable difference is not a finding that survives its own denominator growing, and a conclusion resting on one has to shrink when it does.

Why it fails

A pass count says a model failed. It does not say whether the model tried. Those are different failures, and the trajectory already records which one happened — every cell logs the tools the model actually called.

On ambiguous-anchor at L0, across 34 cells per model:

gemma4:26b never once calls an edit tool. It uses only bash and read: it explores the repository and stops. In seven of the thirty-four it says so plainly — one of them, verbatim:

The instruction was delivered. It is the first user message in the transcript, byte-identical to what the other two models received — checked, not assumed, because “the prompt never arrived” would be the more comfortable explanation.

Everything is in whether the model acts. Attempt rate separates all three (qwen3.8 against qwen3.6, p = 0.003; either qwen against gemma4, p < 0.00000001). But given an attempt, the two qwen models are indistinguishable — 64% against 48%, p = 0.28. So the L0 result above is real and is not what it looks like: qwen3.8:27b does not edit better, it acts more often.

The control that settles it

Same task, same models, same fixture, one rung up at L1 — where the instruction is stated plainly rather than obliquely:

All three models pass 6/6, and all three attempt the edit 6/6. gemma4:26b included.

So gemma4 can make this edit, and does, every time it is asked directly. Change “in runbook.md, change the Staging line” to “Staging needs the force flag when we bounce it”, and it stops registering that anything has been asked of it — thirty-four times out of thirty-four, without one exception. A total, perfectly repeatable failure sitting on top of an intact capability. Which means 0/34 is not, by itself, evidence that a model cannot do something. At this rung the task is not an editing test at all; it is a test of whether the model notices it has been given work.

The ruling written down first

gemma4:26b writes scratch files while it works — test_implementation.py, test_fix.py, repro.py — and the scope check counts each one as a failed cell. Most of its published gap was this rather than a wrong answer.

Two defensible options, and they produce different rosters: treat a scratch write as a behavioural disqualifier, or grade the cell on its answer and count scope violations in their own column. The second was chosen and recorded on 26 September, before the campaign ran, along with the consequence: gemma4:26b's counts would rise and the ordering might move.

They did. Three of its four L1 failures were files created out of scope, not wrong answers, so 17/18 answer-correctness against the 14/18 the unmodified harness scored. One cell is a genuine capability miss.

Writing a repro and testing it is what a competent engineer does; the failure is not cleaning up, which is hygiene rather than capability. The reason to pre-declare is that the same argument made after the numbers would be indefensible — and a column is reversible where a summed failure is not. You can apply a cut later from a column; you cannot recover answer-correctness from a number that has already absorbed it.

Speed, and what the clock is really counting

Quality differs by task rather than by rank. Speed gives an ordering — but only once you know what the clock is counting. Across a full day of trials, decode rate held constant per model while the number of tokens each model chose to write varied by nearly five times. Wall clock tracks output tokens at r ≥ 0.99.

The fastest decoder on the roster is the slowest finisher, by three times. gemma4:26b generates at 207 tok/s and takes 45.6 s, because it writes 4.8× the tokens. qwen3.8:27b writes fewer tokens than the seat and still loses, purely on decode rate. The arithmetic closes in all three cases: tokens divided by decode, plus about 27% for prefill and tool execution.

What is resolvable at n=6, by 95% confidence interval. The seat separates from both on the short task. qwen3.8:27b and gemma4:26b do not separate on it (−2.6 to +4.7 s). gemma4:26b separates from both on the long task. The seat and qwen3.8:27b still do not (−12.9 to +0.6 s). Differences that are not listed as separated are not findings.

The harness was handing over the answer

Every number above was re-measured, because the sandbox the models work in was found to contain the benchmark itself. The carrier bound the whole harness directory next to the fixture, so tasks/*.yaml — each task’s exact grading pattern and all three prompt levels — sat one ls away from the model. It was there to supply a single two-line script.

It is fixed, and L1 and L0 were re-run on the repaired harness. At the level of whole batteries the results reproduced within a cell, so the leak was reachable without being much used.

But that summary was wrong once too, in exactly the way this page is about. Checked per task rather than per battery, one pair moves: gemma4:26b on ambiguous-anchor scored 3/6 with the leak and 0/18 without, p = 0.010. That is the only cell where the leak clearly helped, and it reads as genuine exploitation. Everywhere else it was neutral or made things worse — models burning turns exploring the harness instead of doing the task. An aggregate that reproduces does not license “the defect had no effect”; check at the level the defect could act on.

How many trials, and why

Trial counts are fixed per task by that task's own spread, not per battery. csv-summarize-repair varies by up to 84% of its own mean between identical trials, so at three warm trials the draw picks the winner and picks differently on another night. An n=3 reading once put two models 2.1× apart on it; a paired n=6 run put the same two 1.6% apart.

The obvious suspect was sampling temperature, since 0.8 is high for a battery graded on exact postconditions. It was tested rather than assumed: t0 against t0.8, same model, same task, one cleared card, six warm trials each.

Neither moved detectably. The mean difference is +4.90 s with a 95% interval of −2.8 to +12.6 s, which includes zero, and the variance ratio is F=1.74 against a critical value near 7.15. Temperature is not the lever. At greedy decoding the same task still varied over 4–10 tool calls and 1,615–2,985 output tokens, so the variation is over solution paths, not sampled tokens — which is why n≥6 samples behaviour and cannot be tuned away.

What this does not say

• It does not rank the roster on quality. L2 and L1 are at the ceiling, and at L0 the three models differ by task rather than by position — there is no ordering to state.
• It does not extend the L0 result beyond the pair that was tested. qwen3.8:27b over qwen3.6:35b at L0 is established across four runs; nothing here licenses a general ordering, a claim at another rung, or a claim about other models.
• It does not say the difference is architectural. One model per architecture means “dense wins” and “this particular model wins” cannot be told apart here, and neither is claimed.
• It does not cull anyone. Nothing in this campaign can remove a model that is already seated — disqualification happens at eligibility and at the gate. The seat is chosen from interactive use; the ranking informs that choice and does not make it.
• It does not set a speed cut. Every catastrophically slow case so far was a placement problem that eligibility already removes. A resident-but-slow model is a usability question, governed separately by a 20 tok/s interactive decode floor.
• It does not travel between sessions. The same model, task and trial count read 19.75 s one day and 14.70 s the next, both on a cleared card. The ranges overlap, so it is one population — but the mean moved 26%. Compare arms within a run, not figures across runs.
• It does not screen settings. Which configuration becomes the pin is a separate question with its own history, and two of the four pinned values were justified after the fact rather than derived.
• It does not claim the earlier screening covered these models. It did not: the original span screened an earlier generation, and two of the three models here had never been measured at this rung before.