qwen3.8:27b — dense
66 blocks · no experts · every block active
28.9 tok/s · ctx 131,072 · 63 of 66 layers resident
What a spilled layer actually costs, why a sparse 35B beats a dense 27B on an 8 GB card, and why decode rate does not predict finishing. Three chips, one night. Not a frozen paper.
A model’s speed is not solely an inherent property. Move three of sixty-six layers to the CPU and lose 27.6%. Change what you measure and the ranking inverts. Shrink the card and the larger model wins.
One evening of measurement on an RTX 3090, a Ryzen AI MAX 390 and an RTX 2080. Decode is eval_count / eval_duration read from the daemon, never wall clock — wall clock on a shared daemon includes queueing behind other sessions. Task time is the local-lane L2 battery. Nothing here is UID-verified or registered as a claim.
Two things decide how fast a local model writes: whether all of it fits on the card, and how much of it does work on each token. The first is a budget you can drag. The second is architecture, and no amount of tuning moves it.
A model is a stack of identical blocks. Each block is a layer, and a token passes through every one of them, in order, to produce a single output token. Nothing skips ahead. The count is fixed by the model: qwen3.8:27b has 66, gemma4:31b 61, qwen3.6:35b 42, gemma4:26b 31.
Each layer holds its own slice of the weights, and each does the same two things in the same order: attention, which looks back over everything written so far, then feed-forward, which is where most of the parameters live. That second half is the part a mixture-of-experts model makes sparse. The first half never is.
So when a card cannot hold the whole model, the split is made in whole layers — some on the GPU, the rest in system memory. A readout of 63/66 means three layers' worth of weights are off the card, and every token has to leave the GPU, cross to system memory and come back, three times, before it can be produced. That is why a 4.5% shift in where the weights sit costs 27.6% of the speed: it is not a little more work, it is a detour onto the slowest link in the machine, repeated for every token you read.
Weights are fixed. The KV cache grows with the context you reserve, not with how long your conversation actually is — a short prompt under a large reservation spills exactly as hard. When the two together exceed the card, whole layers get pushed to the CPU.
The 35b is the bigger model on disk and decodes more than twice as fast. Fire a token through both and watch which parameters light up.
66 blocks · no experts · every block active
28.9 tok/s · ctx 131,072 · 63 of 66 layers resident
41 blocks · 256 experts each · 8 fire per token
86.9 tok/s · ctx 131,072 · all 42 layers resident
3.1% of the experts fire. The 35b routes each token to 8 of its 256 experts per block, so despite holding more weight it moves far less of it per token. The 27b is dense — all 27 billion parameters do work on every single token. That is the gap, and no context tuning or card shuffling changes it.
A model does not "think about" your prompt and then answer. It runs one pass to produce one token, appends it, and runs the whole thing again. Everything on this page — the cache that fills your card, the sparse half of a block, the gap between decode rate and finishing time — falls out of that loop.
Why the two differ by 20× per token. Prefill pushes every prompt token through the stack in one batched pass, so the weights are read once and amortised across thousands of tokens. Decode pushes one token through, reads the same weights, and throws them away — then does it again for the next token. Same arithmetic, no one to share it with. That is why decode is bound by memory bandwidth rather than compute, why a sparse model wins there, and why tok/s describes only the second half of the work.
Seconds to finish an agentic L2 task, warm trials. Lower is better. Read each chip against its own two tasks: the ordering changes between them, on every chip.
| Task | Model | seconds | |
|---|---|---|---|
| constant-and-callers | qwen3.6:35b | 13.0 | |
gemma4:26b | 15.0 | ||
qwen3.8:27b | 17.9 | ||
| csv-summarize-repair | qwen3.6:35b | 21.2 | |
qwen3.8:27b | 29.7 | ||
gemma4:26b | 46.0 |
The fastest decoder of all four is last on the long task, by 3×.
| Task | Model | seconds | |
|---|---|---|---|
| constant-and-callers | gemma4:26b | 20.6 | |
qwen3.6:35b | 26.1 | ||
qwen3.8:27b | 57.9 | ||
| csv-summarize-repair | qwen3.6:35b | 53.0 | |
qwen3.8:27b | 60.8 | ||
gemma4:26b | 146.6 |
The 26b is least penalised on the short task and most on the long one, on the same chip. There is no single "this chip is N× slower".
| Task | Model | seconds | |
|---|---|---|---|
| constant-and-callers | gemma4:26b | 40.9 | |
qwen3.6:35b | 59.2 | ||
| csv-summarize-repair | qwen3.6:35b | 120.4 | |
gemma4:26b | 218.6 |
The order flips again: the 35b is slower on the short task and 1.8× faster on the long one. Both are regions, not points — the 26b's trials ran 124.5 / 218.6 / 330.8, the 35b's 108.4 / 136.2 / 120.4. n=3 with a cold trial here, against n=4 warm on the other chips. The 35b passed 2 of 3 on csv-summarize-repair; that miss was a real coding error, not a timeout, on a task whose own header records a ~75% base rate at n=100. The dense 27b has no row on this card: at 2.3 tok/s an agentic task times out rather than finishing slowly. The last bar runs past the axis — true value at right.
The same models, measured as tokens per second. Compare this ordering against the one above.
| Chip | Model | Placement | tok/s |
|---|---|---|---|
| RTX 3090 · ctx 131,072 q8_0 | qwen3.6:35b | 42/42 on GPU | 86.9 |
qwen3.8:27b | 63/66 — 3 on CPU | 28.9 | |
qwen3.8:27b | 66/66 at ctx 98,304 | 42.9 | |
| MAX 390 · ctx 16,384 | qwen3.6:35b | integrated | 60.6 |
gemma4:26b | integrated | 56.4 | |
qwen3.8:27b | integrated | 19.7 | |
| RTX 2080 · ctx 16,384 | qwen3.6:35b | 42/42, 7,094 MiB | 28.2 |
gemma4:26b | 31/31, 7,414 MiB | 20.0 | |
qwen3.8:27b | 16/66, 6,434 MiB | 2.3 | |
| RTX 3090 · ctx 16,384 f16 | gemma4:26b | solo, reference | 171.3 |
qwen3.6:35b | solo, reference | 132.7 | |
qwen3.8:27b | solo, reference | 56.4 | |
gemma4:31b | solo, reference | 33.7 |
Weights stream; they do not have to fit. On the 2080 the 22.6 GB MoE model reports every layer offloaded in 7,094 MiB and beats the 18.6 GB MoE model on the same card. Process memory moves 448 MiB against 14.2 GB of page cache — the weights are memory-mapped and paged from disk, so there is no fixed VRAM-plus-RAM envelope. A dense layer must be resident to compute; an MoE layer only needs the experts a token routes to.
Force layers onto the CPU and the two architectures fail differently. Each curve is scaled to its own full-residency rate; the absolute rates are not comparable across the two. The shape is the point.
gemma4:26bforced offload, 31 layers
| 31/31 | 224.8 | |
| 30/31 | 172.7 | |
| 20/31 | 54.7 | |
| 12/31 | 35.5 |
Slows to a sixth and keeps answering. An earlier sweep floors it near 13 tok/s at 100% CPU.
gemma4:31bVRAM cap, 61 layers
| 55/61 | 12.3 | |
| 50/61 | 8.1 | |
| 45/61 | 6.0 | |
| 40/61 | 4.8 |
At 40/61 the task times out — 2 of 2, re-confirmed. It stops finishing, not just slowing.
Everything above measures speed. That is the smaller half. Six separate suites measure whether a model is any good, and they were built at different times for different questions — they share no fixtures, no task format and no scoring. A number from one is not comparable to a number from another, and none of them is comparable to a throughput figure.
| Category | What it asks | Why it exists |
|---|---|---|
| Task shape | Does the same model succeed more when the task is more structured? Three levels: L0 goal-shaped ("make the tests pass"), L1 file-named, L2 plan-shaped — exact paths, anchored edits, one tool call per step, an explicit success criterion. | The foundational suite; everything else post-dates it. Pass rate rises L0→L2 for every model measured. Shape predicts success better than any "difficulty" claim. |
| Coding implementation | Write working code against a frozen Python substrate — a real nutrition dataset, not a toy. | The reference baseline the other coding-adjacent work defines itself against. |
| Judgment under rules | Preference calls between two pieces of work, and dispute handling that has to stay anchored to a supplied document rather than to the model’s own view. | Scored against answers that were argued out with the people who adjudicate that work for a living — not one author’s guess at a rubric. Held internally; described here, not published. |
| Data comprehension without drift | Reproduce exact facts from a frozen registry extract — implant counts, market shares, an HHI — while preserving product boundaries and refusing to drift into chatbot or clinical-advice framing. | Catches the failure that matters most in practice: a plausible-looking answer with an invented number in it. Scoring is exact literal values, present or absent. |
| Cold-read comprehension | Recover a project's boundary map from five repository documents the model has never seen, with no prior context. | Tracks input-corpus drift as a first-class variable — the corpus grew 63% between epochs, so results across epochs are declared non-comparable rather than quietly pooled. |
| Symbolic constraint compliance | Image generation from one fixed prompt where several meaning-carrying constraints — relative size, pose, arrangement — must all land at once. | A different modality with the same question: does the model satisfy every constraint simultaneously, or only the easy ones? |
Two axes, never pooled. Correctness is one measurement per model; it transfers across chips. Speed is one measurement per machine; it does not — the ordering on this page reverses between chips and between tasks. Re-running a correctness battery on new hardware costs hours and reproduces an answer already in hand. Re-running speed is mandatory.
And the standing rule that keeps them honest: a suite that predates a given measurement contract does not silently inherit it. Each carries its own protocol, its own epoch, and its own note about what it does not establish. Where two runs disagree, the newer one does not automatically win — the one whose configuration was actually recorded does.
gemma4:26b, which is MoE; a CPU-resident MoE layer computes 8 of 128 experts, a dense one computes all of it. The 1.3 ms/layer figure from that curve does not apply to dense models.gemma4:31b. Its expert metadata has not been observed, so "dense" for that model is inference from block arithmetic, not a read.num_ctx, KV type and placement; rows from different groups are not comparable.