Capture native-harness benchmark baselines for qwen2.5 0.5b/1.5b/3b and
qwen2.5-coder:7b (all probed tool-capable; deepseek-r1 and NL2SH reject the
tools field). All cluster at 1-2/12 with high variance; the 7B buys no
pass-rate gain at ~3x latency, so qwen2.5:3b stays the default. The single
biggest score sink across every model is bare tool-call-as-text leaks — a
harness parse gap, the clear next improvement.
Also drop "in your home directory" from the shell prompts: it made literal-
minded models create a home/ subdir (/root/home/a/b/c/...) or use ~/, which
the benchmark itself surfaced. Findings doc + bench README carry the model
comparison table and recommendations (llama3.2:3b / llama3.1:8b for a non-qwen
data point).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
leetcrypt
58d405c518