Skip to content

Agents and LLMsModels and local inferencereference

Qwen3.8-27B for agentic work: speculative decoding and reasoning effort

How the daily-driver Qwen3.8-27B preset got its current shape: MTP speculative decoding disqualified by a decode-degradation bug, the draftless ngram-mod found to degrade the same way, and the xhigh-vs-medium reasoning-effort question settled with a sensor-gated task suite. Picks up where the bench numbers left off; written for anyone serving this model to an agent harness rather than a chat box.

The speculative-decoding and reasoning-effort numbers were measured 2026-08-18 to 2026-08-19 on an RTX 5090 (32 GB), llama.cpp b10362 then b10472, Qwen3.8-27B Q4_K_M (unsloth GGUF), 196608-token context x 2 slots, model-card sampling (temp 1.0, top_p 0.95, top_k 20, min_p 0). The template, quant and thinking-budget sections were measured 2026-09-02 on the same GPU and pin, by then on UD-Q4_K_M at 262144-token context x 1 slot. The NInfer engine section was measured 2026-09-06 to 2026-09-07 on the same GPU: NInfer at upstream commit 487f897, the Qwen3.8-27B NVFP4 .ninfer artifact, 262144-token context on one lane, fp8 KV, --spec mtp --draft-tokens 3 --lm-head-draft. The accuracy numbers (HumanEval, BFCL, the task-suite parity run) were measured 2026-09-08 on the same engine and artifact; the context-ceiling finding the same day. The engine-update and DFlash2 numbers were measured 2026-10-08 on the same GPU, NInfer f76e19c0 against 81c8ce09, the Swift 1.0 NVFP4 .ninfer artifact, 262144-token context on one lane, the preset locked. Every claim below is measured unless marked community-reported; the upstream bugs are cited to their llama.cpp issues.

TL;DR

  • MTP (--spec-type draft-mtp) gives +15% generation on short bursts, then degrades to ~0.5 tok/s within about 10 minutes of agentic task traffic. Restart restores it; the degradation returns. Not usable for loops on llama.cpp; the engine section below has the counterexample.
  • The same model’s MTP sustains on a different engine. NInfer holds a 64% median draft acceptance (45.5-94.4% range) across a whole multi-milestone build-out with decode flat from 30K to 100K of context, which places the failure in llama.cpp’s speculation path rather than in the model’s recurrent state.
  • ngram-mod (draftless, a ~16 MB hash pool over the server’s own history) measured +93-100% generation on a cold pool - and then degraded identically to MTP under real task-suite churn. The only stable config for agentic loops on this model is no speculation at all.
  • reasoning_effort = "medium" is quality-neutral on the task suite (the ceiling tasks fail identically at both efforts) and removes the thinking-binge lottery: xhigh once spent 1308.5 s on a task that also completed in 19.7 s.
  • Bump your llama.cpp pin: b10472 fixes an abandoned-stream slot-parking bug that freezes slots under harnesses that timeout-kill their clients.
  • Capping the thinking budget is the wrong fix: on real work the p99 generation is 9,201 tokens, the p95 is 7,819, and nothing has exceeded 9,564, so any cap low enough to bite cuts reasoning that was converging.
  • On NInfer 81c8ce09 the artifact’s DFlash2 drafter decodes 294.6 tok/s median against MTP’s 188.8 on the same prompts, greedy, at equal HumanEval (0.671) and a 10/10 needle probe at 262144. It needs k8v4 KV to fit 262144 tokens on 32 GB. A sampled A/B of the engine bump showed a 17% decode drop that greedy runs did not reproduce.
  • The GGUF your config names can stop existing. Unsloth deleted every plain K-quant of this model on 2026-08-19 and re-uploaded a dynamic-quant lineup; the preset here kept naming a file that no longer existed for two weeks.
Workloadspec_typeeffortWhy
Agentic loops, benches, unattended runsnonemediumboth speculators degrade under churn; medium kills binges
Interactive coding/chat, babysatdraft-mtp or ngram-modxhigh+15% gen (MTP) or +93% cold (ngram) while the server is fresh; watch for drift
Anything long-lived on b10362none-slot parking on abandoned streams; upgrade
Agentic loops, if you can run NInfermtp (draft 3)mediumspeculation holds under churn on that engine; Qwen-only, sm_120a-only
Same, on NInfer 81c8ce09 or laterdflash2 (draft 7), k8v4 KVmedium+56% short-prompt decode over MTP at equal HumanEval and needle; no churn soak yet

Decode collapses from ~60 tok/s to ~0.5 tok/s, while prompt processing stays fast (~1500 tok/s), MTP draft acceptance stays healthy (0.79-0.99), and the GPU idles at ~100 W draw with 100% “utilization” (idle-clock). Restarting the server restores full speed; the degradation returns after roughly 10 minutes of agentic task churn. Reproduced on both b10362 and b10472.

With speculation disabled, the identical suite sustains 55-66 tok/s indefinitely - one 5433-second, 8-iteration task ran at full speed past request id 216000. That puts the failure in the speculation path rather than the harness or the model, and the next section shows it is not MTP-specific.

Fresh-server synthetic requests never reproduce it. Long prompts (53k tokens), long generations (3500 tokens), streaming, tools, cache-prefix reuse, and long tool-call arguments all decode at 60-104 tok/s. Only a lived-in server crawls, which is why every short benchmark of this model’s MTP looks fine.

Upstream matches both halves of the symptom: acceptance decay over server lifetime with restore-on-restart1, and draft-context corruption after mixed long/short prompt sequences on this exact model2. Both issues open as of 2026-08-19.

MTP survives in one variant preset (qwen38-xhigh) for babysat interactive work, where sessions are short and a restart is cheap.

The slot-parking bug, fixed by the pin bump

Section titled “The slot-parking bug, fixed by the pin bump”

Separately, on b10362: a client that abandons a streaming request (a loop iteration killed by its timeout wrapper, a dead curl) leaves its slot frozen at is_processing=true with n_decoded static, and the slot never frees. With parallel 2, two abandoned streams park the whole server. Verified fixed in b10472: the slot releases about 20 seconds after the client disconnects. Upgrade from b10362 if any harness on the box kills its clients.

ngram-mod: same degradation, different drafter

Section titled “ngram-mod: same degradation, different drafter”

ngram-mod keeps an n-gram -> next-token pool keyed by a linear congruential generator (LCG) hash (~16 MB, constant memory) and drafts from it; the pool is shared across server slots and accumulates accepted tokens over the server lifetime3. It has no draft model and no draft context, which is why it was first adopted here: nothing looked stateful enough to degrade. Within a day the validation suite showed otherwise: under sustained loop-task churn, ngram-mod degrades to the same ~0.5 tok/s crawl as MTP (GPU idling at ~106 W, restart restores, fresh-server probes fine). Priority is fixed in llama.cpp: ngram implementations draft before draft-model implementations when both are configured, each serving sequences the other has no draft for.

Measured against a no-speculation baseline of 73.2 tok/s generation:

variantcold poolwarm poolTTFT p50
defaults (n-match 24, n-min 48, n-max 64)141.1 (+93%)462.1 (+531%)351.0-361.8 ms
small-n (n-min 4, n-max 8, n-match 32)146.3 (+100%)262.6 (+259%)252.7-259.2 ms

Defaults won the warm-pool peak (long drafts land when content repeats verbatim); small-n won TTFT by ~100 ms. The small-n recommendation comes from community testing on consumer GPUs4. Neither variant held up on the real workload, so neither ships, and the table is short-burst evidence only.

The warm-pool number has two limits. The perf harness reuses canned prompts, so a warm pool is maximally matched to them; 462.1 tok/s is the mechanism’s ceiling on repeated prompts and says nothing about an agentic workload. And the pool dies with the server process: any model switch resets it, so the first minutes after a switch run at cold-pool speed.

The shared-mechanism hypothesis for both speculators: every rejected draft forces a rollback of the hybrid Gated DeltaNet recurrent state, and the rollback cost grows with context; at agentic-loop context sizes (24k+ tokens, growing) the rollback dominates until decode crawls, while prompt processing - a different path - stays fast. No-spec never rolls back, which fits the control. Upstream also tracks a separate ngram-mod stuck-loop on failed verification5. If a future pin fixes the state-rollback cost, the perf numbers above justify re-testing; until then this stack runs no speculation on the daily-driver preset.

The hypothesis predicts the failure for any engine running this architecture. One engine does not show it, so the hypothesis is partly falsified.

Both speculator findings above were measured on llama.cpp. If the rollback hypothesis held, any engine running this architecture under agentic churn would crawl the same way. NInfer, a from-scratch C++/CUDA engine for explicitly registered Qwen checkpoints6, does not.

Measured over 336 requests of real agentic coding traffic - the engine driving a four-milestone Rust and TypeScript build-out to all-sensors-green, prompt contexts to 88,985 tokens:

ctx bucketrequestsdecode p50
0-10K18180.0 tok/s
30-50K38141.1 tok/s
50-70K42142.3 tok/s
70-100K37136.5 tok/s

Decode drifts 141 to 136 tok/s across 30K to 100K of context. MTP draft acceptance held at a 64% median (45.5-94.4% range) for the whole run, with zero engine errors and no restarts. On llama.cpp the same speculator on the same model decays to ~0.5 tok/s within ten minutes, and a community measurement thread reports its no-spec decode falling 122 to 69 tok/s as context fills7.

The degradation therefore belongs to an implementation rather than to Gated DeltaNet rollback as such. That narrows where to look in llama.cpp without fixing it, and retires “speculation cannot work on this model under churn” as a conclusion.

NInfer’s coverage is narrow. It serves five explicitly registered Qwen artifact identities and rejects every other checkpoint, its build targets sm_120a only, and each artifact is a single file carrying its own tokenizer and chat template6. Of the eight presets on this box, two map to it and the Gemma and LFM ones never can. It is a second engine for one model family; llama.cpp still serves every other preset.

The day after the speed numbers, accuracy got its own pass: HumanEval, BFCL function-calling, and the sensor-gated task suite the engine was adopted for.

On the six loop tasks (t1-t6: add/fix a Go or TS function, write the tests for it), NInfer at medium went 4/6 - t1, t2, t4, t5 pass in one iteration each (35-81 s); t3 and t6 fail at the eight-iteration cap. Those two are the test-writing tasks and the hardest in the suite: the llama.cpp Q4_K_M baseline goes 4/6 with the same two misses (t3 1/3, t6 0/3 across runs), and every other model benched here fails both. The engine change did not move task quality off its own-weights baseline.

HumanEval pass@1 is 0.598 (greedy, 164 problems, through the proxy), 0.591 with the extra tests. This is the first eval number the harness has produced that did not error at parse time, so it is the reference point future presets run against rather than a delta. BFCL non_live function-calling came back at simple_python 68%, parallel 76.5%, parallel_multiple 73%, multiple 76%, simple_javascript 48%, simple_java 40%. The multi/parallel categories an agent loop uses sit at 73-76%, and the single-file Java and JS stub categories are the weakest. There is no overall BFCL aggregate: the leaderboard aggregator crashed on a null from the irrelevance category, which is a harness bug.

Two accuracy questions remain. Long-context quality, as distinct from the measured flat decode, needs a needle-in-context probe. The tokenizer and hot-swap blockers are fixed (a local HF tokenizer fallback, and a --no-swap mode that probes the resident model at its configured context). The probe then surfaced a sharper limit: the engine rejects a single maxed-out request around 261.5K tokens even though the configured --max-context is 262144 and a simple one-message test serves 261000-262000. The usable ceiling is below the advertised one, and it depends on state and content. A clean retrieval run at that boundary is still pending. Churn stability beyond a single session is also unmeasured: 6.2% of requests in the validation run decoded below 100 tok/s (worst 33.9), clustered at 66-72K context, and the cause is unexplained.

Two configuration problems cost most of a day before any of the above measured cleanly, and neither shows in the harness report.

  • Thinking effort is a per-request field on this engine rather than a serve flag, and the artifact’s template exposes low|medium|xhigh. The OpenAI protocol’s high is parsed and then rejected with reasoning_effort_not_supported, because the engine validates against the template embedded in the artifact rather than translating vocabularies8. A client sending the protocol default gets HTTP 400 on every request.
  • A missing output cap truncates mid-thought. With no explicit maxTokens the harness applied a 16,384 default; at xhigh one tool call produced 35,747 output tokens, so responses were cut before emitting a tool call or any text. The agent exited 0 having changed nothing, twice, which reads as model incapability rather than a ceiling.

Neither is fixable in the preset, because the template ships inside the artifact. They belong in the proxy, which is the one place that fixes them for every client at once.

Setting prefill_chunk = 4096 on the engine cut a 127k-token fresh prefill from 43.4 s to 35.1 s (2.93k to 3.62k tok/s, about 19%), measured 2026-09-14 and set on every NInfer preset since. And the artifact format is versioned against the engine: the 2026-09-18 bump to f76e19c0 reads only v3 .ninfer artifacts, the previous pin reads only v2, so rolling the engine back needs the v2 bytes kept beside the v3 file. Upstream re-released the stock artifact as v3 under the same filename that day, so a file name does not say which format it holds.

The engine pin moved from f76e19c0 to 81c8ce09, 87 upstream commits. The range carries a kernel performance series, a restored accurate SiLU in the NVFP4 SwiGLU kernel (the old pin had an approximate one), a scheduler and context-cache rewrite, and constrained tool calling on by default8. One breaking flag change came with it: --host-state-slots and --host-kv-mib are gone, replaced by a single --host-context-mib budget. The engine rejects the old flags at startup, so a preset still naming them stops the container from booting.

The artifact here is a Qwen3.8-27B finetune (Swift 1.0, a reasoning-trimmed adapter) in NVFP4 with FP8 attention, at 262144-token context and fp8 KV, locked so no other client could contend for the GPU. Both engines ran the same eight short prompts (code, JSON, story, explanation, twice each) plus a 68,545-token prompt.

runsamplingshort-prompt decode medianMTP acceptance median68.5K prefill
f76e19c0sampled193.4 tok/s78.2%6.44k tok/s
81c8ce09sampled160.4 tok/s65.4%7.62k tok/s
f76e19c0greedy190.5 tok/s79.3%6.28k-6.36k tok/s
81c8ce09greedy188.8 tok/s79.8%7.79k-7.86k tok/s

The sampled run looked like a 17% decode regression and was not one. With sampling on, the two engines generate different text, and acceptance depends on the text: on the story prompt alone the old engine accepted 36% of drafts in one run and 84% in another. Greedy decoding makes each engine deterministic (repeated prompts produced identical outputs), and on it the two engines decode within 1% of each other. The prefill gain survives both methods: the long prompt’s time to first token fell from 10.9 s to 8.8 s. Sampled speed comparisons across engine versions need greedy confirmation before anyone acts on them.

The new pin also serves DFlash2, a block-diffusion drafter9, which the artifact already carries alongside its MTP head. At 262144 tokens with fp8 KV it does not boot: the runtime reservation needs 11,152,609,537 bytes against 10,109,321,216 available. With k8v4 KV (FP8 keys, NVFP4 values) it boots with 1.41 GiB free.

greedy, 81c8ce09short-prompt decode mediancode prompt68.5K decode
MTP, 3 drafts, fp8 KV188.8 tok/s196.9-203.2 tok/s153.4-160.0 tok/s
DFlash2, 7 drafts, k8v4 KV294.6 tok/s345.0-350.0 tok/s185.3 tok/s

DFlash2 accepts a smaller share of its drafts (35-73% of 7, against 59-87% of 3 for MTP) but commits more tokens per round, which is the whole of the gain. The drafter was trained by its authors on base Qwen3.8 rather than on this finetune, so the acceptance above is a floor for a matched drafter. Upstream’s own measurement on the stock NVFP4 artifact, at 131072-token context, has the same shape: 348.9 tok/s DFlash2 against 231.7 MTP on structured output, and DFlash2 slightly behind on story text (115.0 vs 132.6)10.

Speculation verified by the target model cannot change greedy output, but the k8v4 KV it needs to fit is a lossy format, so both arms ran HumanEval greedy through the proxy: 0.671 pass@1 (0.659 with the extra tests) for both, on 164 problems, with the suite’s wall time falling from 8 min 44 s to 5 min 7 s. HumanEval is short-context, so a needle-in-context probe covered depth: at 262144 tokens (prompts of about 260,600), five depths, two runs each, the DFlash2 preset retrieved 10 of 10, the same as the fp8 MTP baseline. Prefill at that depth ran 3.75k tok/s, a time to first token near 70 s. What remains unmeasured is churn: DFlash2 has not yet carried a multi-hour agent session.

The pin moved from 81c8ce09 to 1882c6ad, 292 upstream commits. The startup flags were rewritten: --host-context-mib, --device-state-slots and --lm-head-draft are gone, replaced by --host-cache-capacity (it must be 0B under WSL2, which pins about 1 GiB of host memory in total) and --proposal-head optimized. Same Swift artifact, production flags, greedy, run in the order old, new, new, old:

greedy81c8ce091882c6ad
MTP3 short-prompt decode median190.9 / 184.9 tok/s191.7 / 197.4 tok/s
MTP acceptance77.9%78.4%
68.8K prompt time to first token9.0 / 9.4 s9.0 / 8.7 s
HumanEval pass@1 (plus)0.671 (0.659)0.652 (0.646)
needle at 262144, five depths x 210/1010/10

The HumanEval gap is three problems net, and a paired rerun with per-problem results showed what is behind it: the new engine failed 10 problems the old one passed and passed 7 it failed, an exact sign test p = 0.63. A numeric change in the kernels sends greedy decoding down a different path on some problems, in both directions; the net is noise, and a cross-engine bar for greedy HumanEval needs a tolerance, not an equal count.

The new engine reserves more memory at startup. The Swift artifact still fits 262144 tokens with fp8 KV and vision on, about 500 MiB free; the stock artifact (20.0 GiB with its vision tower, against 19.1 GiB) no longer does, whatever the headroom or prefill-chunk setting, and those presets moved to k8v4 KV. On DFlash2 the new engine’s outputs differ from the old one’s and its acceptance on the comparable prompts fell from 51.5% to 47.4%, with decode on those prompts inside the run-to-run spread.

Other ways to run this model, surveyed 2026-10-08

Section titled “Other ways to run this model, surveyed 2026-10-08”

A survey of the alternatives turned up nothing that beats this engine on a single 32 GB card at full context. The candidates, with the reason each was ruled out:

candidateenginewhy not
NVFP4 + MTP on vLLMvLLM 0.27.1 with a backported fixauthor-measured ~160 tok/s on a 509011; needs 4-bit KV to fit, eager mode, one stream only, and stock vLLM garbles output with MTP and that KV format
GSQ-RCO non-uniform GGUFllama.cppstrong quality per byte, but MTP on llama.cpp degrades under agentic churn (above), and one reporter reproduced uniform integer quants of this model failing to converge on long thinking chains, on llama.cpp and vLLM12
groupwise-int .ninfer artifactNInferevals level with NVFP4, but slower: corpus 156 vs 168.6 tok/s and prefill 3.3k vs 12.8k tok/s at 7.7K prompt, upstream-measured10
ternary 27B buildscustom kernelslower tool-calling score than the base model, no 5090 numbers
NVFP4 prefiller over a 1-3 bit GGUF decoder (added 2026-10-20)llama.cpp forkthe authors report +32.5 points MMLU-Pro at 1-bit decode and parity with weight-only at 2-3 bits13; it rescues decoders too small to hold accuracy, and this card already fits the 4-bit model

Strata is an MIT-licensed engine built from llama.cpp parts, first committed 2026-09-24, that runs Qwen3.8-Flash-Next on one consumer GPU14. The model is a 125B mixture-of-experts with 6B active parameters per token, plus a 51B n-gram embedding table15. Strata keeps the experts in system RAM, caches the busiest on the GPU, and leaves the n-gram table memory-mapped on SSD.

Vendor numbers put Flash-Next well ahead of the 27B on agent work (DeepSWE 1.1 58.7 against 42.2) and level on SWE-bench Pro (62.5 against 61.7)15. Those are full-precision scores. The recommended 3.5 bpw quant (IQ3_S) matches the BF16 model on AIME25 and GPQA-Diamond in its quantiser’s own evaluation, and needs 54.8 GB resident for weights alone16. The closest public measurement to this box (one 5090, 64 GB DDR5-4800, IQ3_S, 262144-token context) reports a 106.8 tok/s median decode over 41 real agent requests (range 31.8-170.6) and 3.2k-4.3k tok/s prefill17.

On this box it would lose on speed: about half the DFlash2 decode, with a CPU saturated by expert compute. WSL here sees 47 GB of the host’s 64, short of the IQ3_S working set, so it would also need the WSL memory limit raised. The case for it is quality, if an independent agentic evaluation of the quantised build shows the vendor gap holding, or more host RAM, or context past 262144 (YaRN extends Flash-Next to 1M15). The project was two weeks old at the time of writing, with 470 open issues and pull requests.

A driver update that crashes the engine under WSL

Section titled “A driver update that crashes the engine under WSL”

After a Windows NVIDIA driver update (616.92 to 617.42, the WSL VM left running), every NInfer image exited with SIGILL at initializing CUDA, the old pin and the new one alike, while llama.cpp kept working. The driver was not at fault. WSL reads the driver’s Linux libraries from Windows over 9p, and an engine that had booted earlier on 616.92 left that version’s libnvidia-gpucomp.so pages in the page cache. The replacement file had the identical size, 117,883,752 bytes, so nothing invalidated the cache, and the next process mapped new pages wherever nothing was cached and old ones wherever something was. In gdb the faulting bytes matched the 616.92 file at that offset, not the 617.42 one: new code had jumped into a stale page mid-instruction. llama.cpp does not load that library at startup, which is why it escaped.

Dropping the page cache fixed it without changing the driver:

Terminal window
wsl.exe -d archlinux -u root sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'

wsl --shutdown does the same. Run one of them after any driver update, before the next engine boot.

The A/B ran the sensor-gated task suite with both arms speculation-free, isolating the effort lever.

On quality the two efforts are level. The two ceiling tasks (write-new-tests) fail at both - t3 in 5433 s at xhigh vs 5108 s at medium, t6 in 3000-4079 s historical at xhigh vs 1852 s at medium - so medium reaches the same outcomes and fails in less time.

Speed gate, wall seconds per run (every run passed):

taskxhighmedium
t1 (go, add fn)42.5, 39.0, 44.0, 25.431.4, 19.3, 20.2, 12.7
t2 (go, fix fn)19.7, 1308.5, 35.9, 11.512.5, 19.6, 10.8, 17.2, 25.3, 17.7
t4 (ts, add fn)49.1, 14.2, 38.8, 19.613.1, 10.8, 11.2, 10.6, 9.8, 10.7
t5 (ts, fix fn)11.3, 18.7, 13.6, 12.410.5, 10.3, 10.6, 11.5, 9.6, 11.3

In the xhigh t2 row the same one-iteration task measured 19.7 s in one run and 1308.5 s in another, a single ~80k-token thinking binge at decode speed. Medium’s entire distribution fits in 9.6-31.4 s.

Community reports split: one set runs xhigh unattended overnight with zero failures and sees no reason to drop; another reports endless re-thinking even at medium; a measurement thread puts medium at roughly the previous generation’s default quality18. On this stack’s task suite the hard tasks fail identically at both efforts and the easy ones do not binge at medium. xhigh stays one preset away for babysat work where the extra thinking is worth supervising.

The 35,747-token tool call in the NInfer section came from an unattended loop at inherited xhigh, inside the 15k-40k band the community reports19, and it overran a 16,384-token response cap mid-thought. xhigh’s tail runs into every output ceiling downstream of it.

Mechanics, for llama.cpp: the level goes in as a chat-template kwarg, --chat-template-kwargs '{"reasoning_effort":"medium"}' (compact JSON; the flag word-splits). xhigh is the server default and injects “think carefully” language; community reports put it at 15k-40k thinking tokens per one-shot prompt19. On NInfer it is a per-request field instead, and high is rejected outright - see the engine section.

Effort lowers the average thinking spend and bounds nothing, so llama.cpp’s --reasoning-budget (a hard token cap, -1 for unrestricted) looks like the missing half. An A/B on the fixture suite reports that it has no effect, and that result is wrong.

When the budget is exhausted the server injects the end-of-thinking tag and the model answers from a chain cut mid-sentence, so the cost lands on answer quality.

Generation lengths per request, read off the server’s own timing lines:

workloadrequeststotal generatedp95p99max
4 fixture loop tasks, medium183,902412412412
same tasks, xhigh125,3681,1341,1341,134
one real greenfield build202490,8187,8199,2019,564

The fixture suite cannot answer the question at all. At a 412-token ceiling, any budget worth setting is inert, so an A/B on that suite returns “no measurable effect” whether or not the knob does anything - a false negative that reads like evidence.

On the real task, an 8000-token cap would have cut 9 of 202 requests in a run that passed on its first iteration - reasoning that was on its way to a correct answer. And nothing has exceeded 9,564 tokens, so even a generous runaway guard would never have fired.

The budget was proposed here for a specific incident: one loop task took 1165 s inside a single iteration. The largest generation ever observed is 9,564 tokens, roughly 127 s of wall time at this model’s decode rate, so a 1165 s single-iteration wall is many turns or a stall - a thinking budget would not have touched it. Set a budget when you have evidence of a generation that does not terminate, and set it above the p99 of successful runs. Tuning a cap to make a suite faster truncates generations and reduces the number of tasks it solves.

The counts in the table include thinking and answer tokens together, while the budget caps thinking only, so they are an upper bound on how often a given cap would bind.

From the release-week config threads, measured here and rejected:

  • “Official / recommended” sampling at temp 0.4, top_p 0.90, top_k 15, min_p 0.02. The thread author sourced it from a chat model. The model card’s thinking-mode recommendation is temp 1.0, top_p 0.95, top_k 20, min_p 019.
  • Quantising the KV cache to stretch context (q4_1 main, 73k context on 16 GB). A 16 GB workaround; at 32 GB the model’s full 262144-token context fits on one slot without it, and q4 main KV costs long-context recall. Two slots at that depth do not fit, because this model’s dense KV runs 45.1 KiB per token, so the preset trades width for depth.
  • Quantising the MTP drafter’s KV. Draft KV quantisation only lowers draft acceptance - fewer accepted tokens means slower, never wronger; the target model verifies everything. It also saves almost no VRAM. One Vulkan user reports the opposite ordering, which reads as backend-specific4.
  • Tuning --spec-draft-p-min to raise acceptance rate. One scripted test showed p-min 0.85 lifting acceptance from 76.5% to 92.5% while dropping throughput 7.4%: higher p-min suppresses drafts instead of improving them. Tune for tok/s.

The chat-template fix (2026-08-23, updated 2026-09-02)

Section titled “The chat-template fix (2026-08-23, updated 2026-09-02)”

Qwen’s official Jinja template shipped with real bugs documented and reproduced by the release-week community: enable_thinking=false crashes, multi-turn history gets poisoned with blank \think tags, tool calls crash when arguments arrive as JSON strings (the standard OpenAI format), mid-dialogue system messages get dropped, and on Hermes/agent harnesses the model hallucinates user instructions during thinking2021. The community fix is froggeric/Qwen-Fixed-Chat-Templates, loaded via --chat-template-file.

Deployed here 2026-08-23 at v22.3. Smoke-tested across 13 cases: basic chat, multi-turn memory, tool call + response round-trip, consecutive tool calls (chronological KV preservation), vision, inline <|think_low|> effort steering, mid-conversation system messages, multi-system merge, tool error recovery (two-tier escalation), false-positive disambiguation (code output containing throw new Error), Anthropic thinking field compatibility, and the hallucinated-user-instruction stress test (multi-turn debate). All pass.

v22.4 landed the day after v22.3 was deployed here and went unnoticed for ten days: it adds multi-tool token alignment (dropping a stray newline before <tool_call>) and handling for an assistant message.reasoning field. Measured on the four fast fixture tasks, two runs each, same GGUF, container respawned so the new template loaded: both versions passed 8 of 8, but loop iterations fell from 12 to 8 - every v22.4 run converged on its first attempt, where v22.3 needed two or three on three of them - and total wall time fell from 543.4 s to 452.0 s. The sample is small and wall time on this suite is noisy; the iteration count agrees with the release notes, since a malformed tool call costs an iteration. A template bug can be mistaken for a model failure, so check the template repo on a schedule.

None of this transfers to NInfer. Its artifacts embed their own chat template, resolved at startup from the file rather than supplied by a flag, so there is no --chat-template-file to point at the community fix and no way to apply it short of rebuilding the artifact. Whether those template bugs are present there is unmeasured; the effort vocabulary differs, which is evidence the template is a different one.

When a configured quant is deleted upstream

Section titled “When a configured quant is deleted upstream”

A preset here carries a HuggingFace repo plus filename, which the container uses as a download fallback when the file is absent locally. Nothing re-validated that pair after the first download.

On 2026-08-19 unsloth deleted every plain K-quant of Qwen3.8-27B and re-uploaded a dynamic-quant (UD-*) lineup with an imatrix pass. The same week, both ggml-org gemma-4 repos got the same treatment. Four GGUFs on this box stopped existing upstream while the configs kept naming them. Nothing failed, because the local files happened to survive - had the volume been lost first, the daily driver, the loop engine and the summarisation model would all have been unrecoverable.

HuggingFace’s LFS oid for a file is that file’s sha256, verified here against a 6.7 GB local GGUF, so upstream identity is checkable from the tree API without downloading anything. A weekly job now classifies every preset file as present, renamed, changed, or gone, and copies anything with no upstream copy to off-box storage with a far-end hash check.

The replacement quant also measured better. Both arms ran back to back in one invocation on the same runtime and template, four fixture tasks x 3 runs, with the perf pass re-measured on page-cache-warm files.

passmedian wallmax wallgenerationVRAM peak
plain Q4_K_M12/1252.9 s1165.4 s73.5 tok/s29,897 MiB
UD-Q4_K_M12/1244.8 s124.8 s75.6 tok/s29,217 MiB

UD is 0.6 GB smaller on disk, which leaves more room for KV, and it holds the full 262144-token context with no silent --fit shrink. It is also the only one of the two that still exists upstream, which is what forced the comparison.

Two readings had to be retracted along the way, both from impatience. A first two-run pass had UD looking 63% slower; it did not survive three runs, because the file had been downloaded minutes earlier and its pages were cold. And the first perf sweep reported UD at 30.8 tok/s prompt processing against the incumbent’s 211.8 - a number that sat incoherently beside UD’s lower time to first token. Re-measured warm, both sat at 26-28 tok/s, which turned out to be a bug in the harness rather than a property of either quant (see below).

The daily-driver preset is now: model-card sampling (temp 1.0, top_p 0.95, top_k 20, min_p 0), reasoning_effort = "medium", no thinking budget, no speculation, the community-fixed chat template, unsloth’s UD-Q4_K_M, and the full 262144-token context on one slot. Every release-week optimisation that looked good in a short benchmark - MTP, ngram stacking, KV quant stretches, “recommended” sampling - either degraded under the real workload or was never the author’s to recommend. xhigh + MTP stays one preset away for babysat interactive sessions where the +15% matters and a restart is cheap; if upstream fixes the recurrent-state rollback cost, the ngram-mod numbers above justify a re-test. That config is specific to llama.cpp: on NInfer the same model runs MTP under churn without degrading, so a second preset now carries speculation deliberately rather than as a short-burst curiosity.

  • The synthetic perf suite (TTFT/gen/pp on canned prompts) is noisy across repetitions: the same preset measured 46-96 tok/s gen across days. Trust task-suite wall times and same-session compares; treat any single perf pass as indicative only.
  • Its prompt-processing column was wrong until 2026-09-02. It read prompt_per_second off a roughly 30-token prompt - overhead, not prefill - and took the sample with the best generation rate. It reported 26-28 tok/s while the server’s own timing lines said 2452. Measured properly, on a 17,221-token uncached prompt, this model prefills at 3675.5 tok/s. If a harness reports a prefill rate near its decode rate, suspect the harness.
  • Read medians, not totals or means. One fixture-task run consumed 1165.4 s in a single iteration and moved its arm’s total by 3x while saying nothing about the arm.
  • Two runs is not a sample, and a freshly downloaded GGUF is a cold page cache. Both of this page’s retracted readings came from ignoring one of those.
  • The fixture tasks are a regression gate and cannot measure anything long-horizon. Their largest single generation is 412 tokens against 9,564 on a real build, so context growth, compaction and thinking length are all invisible to them. Measure those on a real scoped task in a worktree.
  • MTP’s short-burst +15% (and medium’s +29% on top of it, 110.5-111.6 vs 85.9-86.5 tok/s) is real but describes the first minutes of a server’s life - exactly the regime the degradation finding disqualifies for loops.
  • ngram-mod’s warm-pool ceiling is inflated by the harness. The cold-pool +93-100% is the number to use.

The bench harness is the llmc bench module of the llmc stack: llmc bench perf --presets a,b for the synthetic suite, llmc bench tasks --presets a,b --tasks t3,t6 --runs 1 for the sensor-gated suite. Preset A/B variants need distinct GGUF filenames because presets deduplicate by model id (derived from the filename stem) - a hardlink costs zero disk. The p5 bench script embeds a /slots watchdog that respawns the server when a slot stops making progress, which is what made the long suite runs survivable while the degradation was live. The proxy the runs go through - mode swaps, drain-before-swap, and the preset lock every measurement here depends on - is described in Scheduling one GPU between LLM, image and training work.

  1. ggml-org, “MTP draft acceptance collapses over server lifetime,” llama.cpp issue #27151. https://github.com/ggml-org/llama.cpp/issues/27151 ↩

  2. ggml-org, “MTP breaks long/short inference (Qwen3.8-27B),” llama.cpp issue #27296. https://github.com/ggml-org/llama.cpp/issues/27296 ↩

  3. ggml-org, “Speculative decoding,” llama.cpp docs. https://raw.githubusercontent.com/ggml-org/llama.cpp/master/docs/speculative.md ↩

  4. r/LocalLLaMA, Qwen3.8-27B config thread and comment discussion (small-n guidance, draft-KV quantization), August 2026 (mirror). https://shoftech.com/question/qwen-3-8-27b-pushing-1m-tokens-on-16gb-vram-whats-your-llama-cpp-config-for-similar-agentic-coding-73k-context/ ↩ ↩2

  5. ggml-org, “server: stuck-loop escape for ngram-mod (WIP),” llama.cpp PR #25819. https://github.com/ggml-org/llama.cpp/pull/25819 ↩

  6. Neroued, “NInfer,” GitHub. https://github.com/Neroued/ninfer ↩ ↩2

  7. r/LocalLLaMA, long-context decode-decay measurements for llama.cpp against vLLM and NInfer, August 2026. ↩

  8. Neroued, “HTTP serving,” NInfer docs. https://github.com/Neroued/ninfer/blob/master/docs/serving.md ↩ ↩2

  9. z-lab, “Qwen3.8-27B-DFlash2,” Hugging Face. https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2 ↩

  10. Neroued, “Qwen3.8-27B performance,” NInfer docs (measured 2026-09-28 at 7f6aafed). https://github.com/Neroued/ninfer/blob/81c8ce093b2c1646a87566a8e59d807fcf0ec95c/docs/performance/qwen3.8-27b.md ↩ ↩2

  11. MiaAI-Lab, “Qwen3.8-27B NVFP4 on RTX 5090,” GitHub. https://github.com/MiaAI-Lab/Qwen3.8-27B-NVFP4-RTX-5090 ↩

  12. unsloth, “Uniform GGUF quants silently break Qwen3.8-27B’s deep thinking,” Hugging Face discussion #135. https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/discussions/135 ↩

  13. A. Panferov et al., “Disaggregated Quantization: Specializing LLM Prefill and Decode,” arXiv. https://arxiv.org/abs/2609.26333 ↩

  14. Niko1221, “Strata,” GitHub. https://github.com/Niko1221/Strata ↩

  15. Qwen, “Qwen3.8-Flash-Next model card,” Hugging Face. https://huggingface.co/Qwen/Qwen3.8-Flash-Next ↩ ↩2 ↩3

  16. ISTA-DASLab, “Qwen3.8-Flash-Next-GSQ-RCO-GGUF,” Hugging Face. https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF ↩

  17. Wisely Chen, “Strata + Qwen3.8-Flash-Next on an RTX 5090: 1M context,” 2026-10-01. https://en.wiselychen.com/en/strata-qwen38-flash-next-rtx5090-1m-context/ ↩

  18. Isaak Tsalicoglou, “Qwen3.8-27B: slower tokens, faster and better results,” OVERBRING Labs. https://overbring.com/blog/2026-08-17-qwen3-8-27b-wall-clock/ ↩

  19. Qwen, “Qwen3.8-27B model card,” Hugging Face. https://huggingface.co/Qwen/Qwen3.8-27B ↩ ↩2 ↩3

  20. froggeric, “Qwen-Fixed-Chat-Templates,” Hugging Face. https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates ↩

  21. r/LocalLLaMA, “I fixed the Qwen 3.8 chat template bugs” (334 points), August 2026. ↩