Benchmarking local models for agent work

How the local LLM stack picks its models: a bench harness that runs four suites against OpenAI-compatible presets, and the numbers that decided which model runs agent loops, which one answers interactively, and which ones fit the small GPU on the server. Written for anyone benchmarking GGUFs for agent-shaped work rather than leaderboard-shaped work.

All numbers were measured 2026-08-15 to 2026-08-17 on an RTX 5090 (32GB), llama.cpp b10362, Q4_K_M quants, through the stack’s own proxy - so every measurement carries the sampling params the model serves with. Small-track numbers are the same harness on the same GPU; the target hardware (a 3080 Ti) is called out where it matters. Community-reported behaviours are marked as such and were not re-measured here.

TL;DR

• Agent loop engine: Gemma 26B-A4B MoE. 199 tok/s decode, ties the 27B dense models on real loop tasks, and a doomed task costs 10-45 minutes instead of 75.
• Interactive coding and thinking: Qwen3.8 27B with multi-token prediction (MTP) speculation on - gen +15.1%, prompt processing +38.1%, time to first token (TTFT) -20%, still fits the 196k-context production config at 31.9GB.
• Small track (server GPU): gemma4-12b ties the qwen35-9b incumbent at 0.944 case-hit rate with fewer wasted steps. Deploy-fit check pending.
• Qwen3.8 defaults to reasoning_effort: xhigh, which is 15k-40k thinking tokens per nontrivial prompt; medium is a couple thousand. Every number below was measured at the xhigh default.
• The Berkeley Function Calling Leaderboard (BFCL) took five harness fixes to run against a local endpoint; the aggregate writer still crashes on a locally-registered model’s absent latency column, but the per-category scores now land (2026-09-08), so the suite yields a function-calling read.

Which model for which job

Role	Model	Why
Unattended agent loops	Gemma 26B-A4B MoE	2.7x the decode speed of the dense options at equal task-suite score; failure is cheap
Interactive coding / thinking	Qwen3.8 27B + MTP	Best HumanEval (0.451 vs 0.293 / 0.116), 196k context, 85 tok/s with speculation
Chat / vision	Qwen3.8 27B + MTP	Same preset; vision smokes pass, 3x the context of the dense Gemma
Server-side research agent	qwen35-9b, pending gemma4-12b	Tie on hit rate; gemma4-12b wins on steps (1.67 vs 2.30) and forced-final rate
Eliminated	qwen36-moe, lfm25-8b, qwen35-4b	Fast-but-shallow (196 tok/s, pp 57.9), protocol-fragile (0.722 hit), small-but-sloppy (0.870)

The harness

Presets are TOML files; the proxy (Go, model-proxy-go) live-reloads them and enforces a persistent lock with TTL + FIFO queue so a benchmark and an interactive session never share the GPU. It also drains in-flight requests before a model swap (no killed streams) and can serve a request in place on the resident model when it advertises the requested capability. Throwaway A/B presets (e.g. the context-occupancy sweep’s ctx-sweep-<n>) register in memory via POST /v1/presets rather than landing in the live preset dir. Every record in the trend store carries the llama.cpp build and a hash of the preset, so a tuning change shows up as a new provenance rather than silently rewriting history. Suites:

• perf - TTFT p50/p95, gen and prompt-processing tok/s, VRAM; the cheapest suite.
• tasks - six sensor-gated agent tasks (scoped edits, fixes, write-new-tests) run through the real loop engine with exit-code sensors. It measures the work the loop engine does all day.
• eval - HumanEval via evalplus¹, HellaSwag via lm-eval, BFCL via bfcl-eval², in one container.
• needle - long-context retrieval: fills a prompt to a fraction of the context budget, splices a codeword at a depth, scores whether the model finds it. Falls back to a local HF tokenizer for engines without a /tokenize endpoint (NInfer), and has a --no-swap mode for single-resident engines that cannot hot-swap an ephemeral preset.
• gumshoe - an 18-case research-agent protocol suite against stub tools; measures case-hit rate and JSON protocol validity. Used for the small models that serve the research stack.

The numbers

Perf (b10362, RTX 5090):

preset	TTFT p50	gen tok/s	pp tok/s	VRAM MiB	ctx	slots
loop (Gemma 26B-A4B MoE)	160.7	199.2	178.7	22536	196608	2
gemma4 (Gemma 31B dense)	363.2	63.0	306.8	26689	65536	1
qwen3.8 (27B dense)	373.2	74.0	19.7	27259	196608	2
qwen3.8 + MTP	297.1	85.2	27.2	30072	196608	2
qwen36-moe (eliminated)	202.1	196.0	57.9	27078	163840	1

Task suite, 6 tasks x 3 runs each:

preset	scoped edits	fixes	write-new-tests	total
loop	6/6	6/6	0/6	12/18
gemma4	6/6	6/6	0/6	12/18
qwen3.8	6/6	6/6	1/6	13/18

Every preset one-shots scoped edits and every preset stalls on write-new-tests, so that task is the current suite’s ceiling and does not separate the models. With the score tied, the loop-engine pick falls to economics: a task that will never pass burns 10-45 minutes on the MoE versus up to 75 on qwen3.8.

HumanEval, greedy, through the proxy:

preset	pass@1
loop (MoE)	0.116
gemma4 (dense)	0.293
qwen3.8 (dense, thinking)	0.451

Gumshoe small track, 18 cases x 3 repeats:

preset	case-hit	json valid	mean steps	forced final
qwen35-9b (incumbent)	0.944	0.983	2.30	0.111
gemma4-12b	0.944	0.917	1.67	0.093
qwen35-4b	0.870	0.930	2.43	0.259
lfm25-8b	0.722	0.854	3.02	0.389

One case (g15-chain, a multi-tool dependency chain) is 0/3 for every preset, so it separates nothing until a preset passes it.

MTP speculative decoding

Qwen3.8 ships a multi-token-prediction head that llama.cpp can run as a self-draft: the main model’s MTP head proposes continuations and the full model verifies them, no separate drafter file needed³. One preset line (spec_type = "draft-mtp") gives gen +15.1% and prompt processing +38.1% at +10.3% VRAM. The context ceiling held (196608 x 2 slots fits at 31948 MiB, just under the card), and the accuracy guard passed 4/4 loop tasks at 12-31s each. It is now on by default in the qwen3.8 preset.

Superseded 2026-08-19

MTP was later found to degrade under agentic churn (decode collapses within ~10 minutes of loop traffic). The preset moved first to draftless ngram-mod, which degraded identically within a day, and then to no speculation at all plus reasoning_effort = "medium" - the follow-up measurements are in Qwen3.8-27B for agentic work.

Chat template fixed 2026-08-23, updated 2026-09-02

Qwen’s official Jinja template was found to have bugs (empty-think poisoning, tool-call crashes, dropped system messages). All presets now use froggeric/Qwen-Fixed-Chat-Templates, on v22.4 since 2026-09-02. Details in the agentic tuning writeup.

Quant changed 2026-09-02

The numbers on this page are plain Q4_K_M quants. Unsloth deleted every plain K-quant of Qwen3.8-27B on 2026-08-19 and re-uploaded a dynamic-quant lineup, so the Qwen3.8 presets now run UD-Q4_K_M - smaller, marginally faster, and unlike its predecessor still published. The A/B and the upstream-deletion story are in the agentic tuning writeup.

The MTP spike surfaced one proxy constraint: presets are deduplicated by model id, which derives from the GGUF filename - two presets pointing at one file crash-loop the proxy at startup. An A/B preset for the same weights needs its own filename; a hardlink gives that at zero extra disk.

Running Qwen3.8 in practice

The model’s reasoning is steerable, and the steering matters more than any sampling param. Community findings⁴, consistent with what the bench saw:

• reasoning_effort is prompt-level soft steering with three trained levels. xhigh (the default) injects “think carefully, validate assumptions, consider alternatives” language; medium injects nothing; low injects a brevity instruction. On llama.cpp the level goes in as chat-template kwargs: --chat-template-kwargs '{"reasoning_effort": "medium"}', or the same key per request.
• The level swing is 10-20x of thinking tokens. Community reports put xhigh at 15k-40k thinking tokens on a one-shot coding prompt, medium at a couple thousand - less than the previous generation. Every number in this doc was measured at the xhigh default: the humaneval 0.451 is an xhigh number, and the 74-token/s effective speed is an xhigh speed. Medium is the interactive default: re-measured, it was quality-neutral on the task suite and avoids xhigh’s unpredictable thinking length (see Qwen3.8-27B for agentic work).
• It “re-reads” files by decoding them. In agentic harnesses the model will say “let me read X” and then reproduce a file it was already given, verbatim, inside its reasoning trace - a learned attention re-weighting behaviour (a cited copy in the trace gets more weight than the distant original), not a tool call. It reportedly lowers code hallucination, but it burns decode tokens at decode speed and inflates the context; effort level does not disable it.
• A reasoning budget truncates the trace. llama.cpp’s --reasoning-budget N cuts it mid-thought and appends a budget message; the effort levels are trained-in and end cleanly. Truncation risks the degraded-answer failure mode. The opposite failure mode is worse for loops: a small max_tokens plus a long trace yields an empty final response, which breaks agentic loops that expect content. Size max_tokens for the trace plus the answer.
• Thinking traces interact with compaction. In a harness that compacts context, long traces crowd out earlier file contents and tool outputs. With 196k of context and 2 slots this is comfortable; on smaller budgets it is the first thing to watch.

Reading the numbers

Absolute HumanEval scores here run far below the published model-card numbers, because greedy decoding through the proxy uses the preset’s serving params rather than a tuned eval config. The ranking is the usable signal, and it matches the task-suite ordering.

The MoE-versus-dense split decided the picks. The MoE gives 199 tok/s and cheap failure but is visibly weaker on synthetic codegen (0.116 vs 0.293 for its own dense sibling). The dense thinking model is the slowest thing on the card at 74 tok/s unaccelerated, and still the right interactive model: on the identical 164-problem codegen run it took 34:48 against the MoE’s 11:37, and the quality shows up exactly where the loop engine never goes (0.451 on codegen it was never asked to do unattended).

Two things generalise: measure models against the job’s own protocol (the task suite caught nothing the perf suite predicted, and vice versa), and treat unattended-loop economics as a first-class metric, cost of failure as well as cost of success. None of the absolute numbers generalise off this rig.

The bfcl-eval landmines

BFCL did not produce a number at first (the 2026-09-08 update below has the per-category scores). Getting bfcl-eval 2026.3.23 to run at all against a local endpoint took five fixes, and the sixth crash lives inside its leaderboard CSV formatter. In the order they appeared:

1. Uninstallable dependency pin. bfcl-eval pins faiss-cpu==1.11.0, which PyPI no longer carries - install with --no-deps and relax faiss by hand.
2. A tree-sitter ABI split. evalplus requires tree-sitter>=0.22 and calls Language(capsule); bfcl’s java parser calls the legacy 2-arg Language(capsule, "java"), which only exists on 0.21.x⁵. No single install satisfies both - the answer is two virtualenvs in one image.
3. An undeclared transitive import. qwen_agent imports soundfile at module top without declaring it; with --no-deps that is a crash on first invocation.
4. A renamed test taxonomy. The ast category is gone; non_live is the same static set. The live/web_search/agentic categories need external API keys - skip them for local models.
5. A hard model registry. --model resolves through a 175-entry config map and KeyErrors on anything else, and the evaluate path applies an underscore-to-slash conversion meant for vendor model ids, which mangles GGUF stems like Q4_K_M. A sitecustomize shim registers local model ids (both spellings) to the generic OpenAI handler at interpreter start.
6. An open leaderboard-writer crash. Scoring completes, then the leaderboard writer crashes multiplying a None latency column - locally registered models carry no cost or latency data. The per-category score files are written before that step, so the fix is parsing those instead of the aggregate CSV.

The work stopped here because the open model decisions were already answered by the other suites, and the gumshoe suite measures tool-use protocol validity - closer to how these models are driven than BFCL’s AST matching. The fixes are committed, so the work can resume from here.

Update, 2026-09-08: BFCL now produces per-category numbers. The NInfer engine was run through the full non_live collection: simple_python 68%, parallel 76.5%, parallel_multiple 73%, multiple 76%, simple_javascript 48%, simple_java 40%. Landmine 6 still applies as written - the leaderboard aggregator crashes multiplying the irrelevance category’s None latency - so the aggregate CSV is still absent and the record carries the per-category scores only. Those categories are the ones an agent loop exercises, so the suite is now usable as a function-calling check. See the Qwen3.8 tuning notes for the full accuracy pass.

Reproducing

Everything lives in the llm-compose repo (public). llmc bench perf|tasks|gumshoe --presets ... for the fast suites; llmc bench eval --presets ... --humaneval for evalplus.

Every suite also takes --external URL --model-id ID, which skips the preset lock and mode switch and measures whatever OpenAI-compatible endpoint you point it at. That arm exists because a preset is a llama.cpp concept: evaluating a different engine meant either inventing a fake preset for it or measuring outside the harness and losing the store. Results still land in bench/results/runs.jsonl with a hash of the flags standing in for the preset hash, so an external arm is comparable against preset runs in the same store. It is how the NInfer numbers in the Qwen3.8 tuning notes were taken. The full harness writeup, including the dual-venv Dockerfile layout and the sitecustomize shim, is in docs/reference/eval-harness.md; the decision record is docs/plans/2026-08-16-p3-matrix-results.md.

Claim	Status	How it was checked
Perf numbers (all presets)	measured	llmc bench perf, runs 20260816-091630 / 20260817-033645
Task-suite matrix	measured	6 tasks x 3 runs x 3 presets, sensor exit codes
HumanEval pass@1	measured	evalplus greedy, 164 problems per preset, xhigh effort
Gumshoe rates	measured	18 cases x 3 repeats per preset
MTP deltas and fit	measured	perf A/B + ctx-ceiling probe + 4/4 guard tasks
External-endpoint arm	measured	llmc bench perf --external against a non-preset engine, 2026-09-06
HumanEval below model-card values	asserted	consistent with greedy-through-proxy config; not re-derived per model
reasoning_effort levels and token swing	community-reported	Qwen/Unsloth discussion and r/LocalLLaMA, 2026-08; not re-measured
Verbatim re-read behaviour	community-reported	same; not observed systematically here
BFCL landmines 1-5	measured	each reproduced and fixed in the image, fixes re-verified in-container
BFCL landmine 6	measured	crash reproduced twice and still open; since 2026-09-08 the per-category score files are read instead, so only the aggregate is absent

References

1. EvalPlus, “EvalPlus: rigorous evaluation of LLM-synthesized code,” GitHub. https://github.com/evalplus/evalplus ↩

2. Shishir Patil et al., “The Berkeley Function Calling Leaderboard (BFCL),” GitHub. https://github.com/ShishirPatil/gorilla ↩

3. ggml-org, “llama.cpp,” GitHub. https://github.com/ggml-org/llama.cpp ↩

4. Unsloth, “Why the reasoning so much?” (Qwen3.8-27B-GGUF discussion 20), Hugging Face. https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/discussions/20 ↩

5. tree-sitter, “py-tree-sitter,” GitHub. https://github.com/tree-sitter/py-tree-sitter ↩