The first harness changes to move the benchmark off the 1/12 noise floor, on
the model that needs it most (qwen2.5:0.5b: 1/12 -> 4/12, 3/12).
- complete_with_tools now decodes the tool loop at temperature 0 (scoped; chat
keeps default sampling). At Ollama's default 0.8 the weak model sampled away
from the tool-call format into prose/fabrication; the nudge prompt changes
between turns so temp 0 still escapes a failed state on retry.
- Greedy decode made 0.5b's leak deterministic, exposing its real shape: not a
JSON object with a name key, but the name in a <tools> tag and the args in a
SEPARATE object — <tools>write_file</tools>{"path":…} — ~5 of 12 tasks/run.
_NAMED_TAG pairs the tag-name with the following args object, gated on the
known tool set so it still can't fabricate an action.
- Bridge recovers a ```bash block narrated in prose as a run_shell call,
non-destructive only (FENCE_DESTRUCTIVE guard); fires on prose-leak turns,
no-op where the model emits structured calls.
Ablation on 0.5b: structured-JSON-only 0/0 -> fenced+temp0 2/0 -> +split-tag
4/3. The lift is concentrated on the weakest model by design — a 3B emits
proper calls and fails on capability/content (unchanged at 1/12), which no
parser can fix. All recovery paths unit-checked for the positive shapes and
the negatives (prose / unknown tool / destructive block) they must ignore.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Generalise OllamaProvider._extract_text_tool_calls to recover a tool-call
JSON object regardless of how a small/quantized model wraps it — qwen's
<tool_call> tags, bare JSON, ```json fences, alternate tags (<tools>,
<function_call>), OpenAI {"function":{…}} nesting, and parameters-vs-arguments.
A new _coerce_call gates recovery on the known tool-name set from the tools
schema, so a stray JSON blob in prose (or a hallucinated make_dir) can never
be coerced into an action. 11-case unit check: 8 leak shapes recover, 3
negatives (prose / unknown tool / random config JSON) ignored.
Benchmark verdict (honest): this does NOT move the weak-CPU-model pass rate
— 3b went 2/1/0 of 12 across three passes (baseline 1/12, noise), 0.5b went
0/0 (baseline 1/12). A direct /api/chat probe shows the hypothesis was wrong
about the FORM of the leak: the weak models emit either malformed structured
tool_calls (write_file content:null) or a fenced bash block in prose with no
tool call at all — not JSON-as-text. The structured-JSON recovery is still a
correct, safe hardening for any model that does leak JSON; the real
weak-model lever (parse ```bash fences -> run_shell) is documented as an
explicit safety decision, not folded in here.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Capture native-harness benchmark baselines for qwen2.5 0.5b/1.5b/3b and
qwen2.5-coder:7b (all probed tool-capable; deepseek-r1 and NL2SH reject the
tools field). All cluster at 1-2/12 with high variance; the 7B buys no
pass-rate gain at ~3x latency, so qwen2.5:3b stays the default. The single
biggest score sink across every model is bare tool-call-as-text leaks — a
harness parse gap, the clear next improvement.
Also drop "in your home directory" from the shell prompts: it made literal-
minded models create a home/ subdir (/root/home/a/b/c/...) or use ~/, which
the benchmark itself surfaced. Findings doc + bench README carry the model
comparison table and recommendations (llama3.2:3b / llama3.1:8b for a non-qwen
data point).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add bench/ — a 4-category × easy/medium/hard task matrix (shell, code, git,
multi) and a runner that drives the live TUI over tmux and grades each task by
a `podman exec` verify snippet (exit 0 == PASS), never by the model's
self-reported summary (which the weak CPU model fabricates). Tasks run in the
agent's real cwd with bare filenames so the suite measures task completion, not
the model's absolute-path discipline. Completion is detected off the viewport-
independent `is thinking…` footer (the TUI is full-screen, so capture-pane
scrollback is not chat history).
First baseline (qwen2.5:3b): 2/12 PASS, high variance. Surfaces the next
harness-addressable improvements — `<native>` tag leakage and bare
tool-call-as-text — now measurable against this suite. Findings doc updated.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace the overloaded "text + no tool call = done" terminator that made the
weak CPU model stall mid-task or give up after a failing command. Termination
is now a structural `DONE:` text sentinel; a text-only turn is resolved by an
output-aware verdict (DONE: marker / unresolved non-zero exit / no action /
filler language) and re-prompted with an exit-code-aware nudge, bounded by
MAX_NUDGES on top of max_turns. On exhaustion the summary is honest rather than
echoing the model's false "run successfully" — it reports when no tool ran or a
command exited non-zero. Live-validated on qwen2.5:3b: the multi-step stall is
fixed (proj3 completes end-to-end, ground-truth confirmed); the nudge fires on
a 126; residual give-up is model-bound. 23 offline unit assertions pass.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Live 3B-vs-7B command-entry results, observed failure modes (early stall,
give-up-on-error, tool hallucination), Goose/opencode loop-termination
research, and the output-aware dynamic-nudge-loop design to implement.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
leetcrypt