Pulled and benchmarked three more tool-capable CPU models looking for a better
default. All score 0/12 (vs qwen2.5:3b at 2/12): in the multi-turn agent loop
they leak the positional-in-tags dialect (<tools>run_shell 'cmd'</tools>) the
parser can't recover, even when they emit clean structured tool_calls on a
single-turn probe; smollm2 and mistral also wedge into repeating summaries.
qwen3:4b could not be pulled — Ollama 0.3.9 is too old (HTTP 412), same as
granite3.1-dense:2b. Upgrading Ollama is the highest-leverage next step to test
the qwen3/granite3.x generation. qwen2.5:3b remains the default.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add tracked baselines for two additional CPU models under the optimized
harness (split-tag recovery + greedy decode). Both land at 1/12 — they emit
proper structured calls and fail on capability/content, not parse, confirming
the parser lift is concentrated on the weakest model (0.5b). granite3.1-dense:2b
is incompatible with the installed Ollama version.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Capture native-harness benchmark baselines for qwen2.5 0.5b/1.5b/3b and
qwen2.5-coder:7b (all probed tool-capable; deepseek-r1 and NL2SH reject the
tools field). All cluster at 1-2/12 with high variance; the 7B buys no
pass-rate gain at ~3x latency, so qwen2.5:3b stays the default. The single
biggest score sink across every model is bare tool-call-as-text leaks — a
harness parse gap, the clear next improvement.
Also drop "in your home directory" from the shell prompts: it made literal-
minded models create a home/ subdir (/root/home/a/b/c/...) or use ~/, which
the benchmark itself surfaced. Findings doc + bench README carry the model
comparison table and recommendations (llama3.2:3b / llama3.1:8b for a non-qwen
data point).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add bench/ — a 4-category × easy/medium/hard task matrix (shell, code, git,
multi) and a runner that drives the live TUI over tmux and grades each task by
a `podman exec` verify snippet (exit 0 == PASS), never by the model's
self-reported summary (which the weak CPU model fabricates). Tasks run in the
agent's real cwd with bare filenames so the suite measures task completion, not
the model's absolute-path discipline. Completion is detected off the viewport-
independent `is thinking…` footer (the TUI is full-screen, so capture-pane
scrollback is not chat history).
First baseline (qwen2.5:3b): 2/12 PASS, high variance. Surfaces the next
harness-addressable improvements — `<native>` tag leakage and bare
tool-call-as-text — now measurable against this suite. Findings doc updated.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
leetcrypt