hack-house/docs/findings-native-harness-2026-06-10.md

Native harness — live command-entry findings + nudge-loop design

Date: 2026-06-10 Context: Evaluating the native tool-calling harness (cmd_chat/agent/bridge.py, _run_native) on its ability to drive shell commands into the shared podman/Kali sandbox, on a CPU-only box (no GPU). Two prior fixes were live-validated here and committed in a0aa14d: §3 PTY-sentinel (_run_shell_in_pty) and NATIVE_CONTEXT=4.

Test setup

• Fresh room, podman sandbox (hack-house container), /grant ai drive.
• Models compared: qwen2.5:3b (1.9 GB, prior default) and qwen2.5-coder:7b (4.7 GB, pulled for this test). deepseek-r1:latest and westenfelder/NL2SH were ruled out — both reject Ollama's tools field (HTTP 400 "does not support tools"), so they degrade to the simple injector and can't exercise the native loop.
• Each task verified against ground truth (podman exec into the container), not just the chat/PTY surface.

Results: qwen2.5:3b vs qwen2.5-coder:7b

Behaviour	qwen2.5:3b	qwen2.5-coder:7b
Single command (whoami)	PASS	PASS
write+run (hello.sh date+hostname)	PASS, accurate summary	PASS
Multi-step (mkdir→write→list)	FAIL — stalled after step 1 ("ok, let's proceed to the next step")	PASS — chained make_dir→write→read, completed
Recover from mid-task error	FAIL — gives up	INCONSISTENT — recovered from [unknown tool make_dir], but on greet.sh skipped chmod, hit exit 126, then stopped
Tool-schema discipline	clean	hallucinated a make_dir tool (not in schema) — wasted a turn ([unknown tool make_dir], bridge.py:597)
Final summary quality	vague ("I will proceed…")	empty → (done) fallback, or raw advice text
Latency (CPU, no GPU)	~20–40 s/task	~2–4× slower, 30–90 s/turn

Failure modes observed (both sizes)

1. Early termination on multi-step (3B dominant): the model emits filler text with no tool call mid-task; the loop reads "text + no tool call = done" and stops. The remaining steps never run.
2. Give-up on unresolved non-zero exit (both sizes): after a failing command (exit 126/127), the model emits a summary/advice instead of fixing the cause. greet.sh left at mode 644 (never chmod'd) despite the task saying "make it executable".
3. Tool hallucination (7B): invented make_dir; the schema only has run_shell/write_file/read_file.

Root cause in code

_run_native, bridge.py:739–741:

if not calls:
    final = (text or "").strip()
    break        # ANY text-without-toolcall ends the task

NATIVE_SYSTEM (line 111–113) also tells the model to signal completion this exact way. So one signal — "text, no tool call" — is overloaded to mean BOTH "done" and "stalling/thinking". Disambiguating it is the fix.

Takeaway: a bigger model buys multi-step persistence (the 3B's main flaw) but does NOT eliminate the give-up-on-error mode, and costs heavy latency. A nudge loop that keys off exit codes (not just filler text) earns its keep at both sizes.

Research: how Goose & opencode structure loop termination

(Source read directly from clones; key files cited.)

• Goose (crates/goose/src/agents/agent.rs): tracks no_tools_called (line 1801). When true it does NOT just stop (lines 2232–2316) — in structured/goal modes it injects a continuation nudge (FINAL_OUTPUT_CONTINUATION_MESSAGE = "You MUST call the final_output tool NOW…") and loops. Completion is an explicit terminal tool (recipe__final_output, final_output_tool.rs). Vanilla chat with no recipe/goal does fall through to text-as-done, but every structured path overrides that. Bounded by max_turns.
• opencode (packages/opencode/src/session/prompt.ts:1156–1183): exits only when finish is a real stop reason AND it independently re-derives that there are no pending tool calls from the parsed message parts — comment: "Some providers return 'stop' even when the assistant message contains tool calls." Hard step cap (maxSteps) with a wind-down message (MAX_STEPS, prompt/max-steps.txt); structured mode forces toolChoice: "required".
• Canonical / Cline-Roo: weak-model harnesses favour an explicit completion action (attempt_completion/submit) over trusting "empty tool_calls = done".
• Recommendation from research: go structural, not filler-heuristic ("a losing arms race"); nudge on text-without-completion; a hard cap is the only unconditional exit; derive "did a tool get called" from parsed parts, not finish_reason (qwen on CPU is unreliable there — it leaks tool calls as <tool_call> text in content; already handled by OllamaProvider._extract_text_tool_calls).

Design: dynamic nudge loop (output-aware, structural terminator)

Structural termination via a DONE: text sentinel rather than a 4th tool. Divergence from the research's "add a done tool", justified for THIS model: qwen-on-CPU already mis-emits structured tool calls (leaks as <tool_call> text; the 7B even hallucinated make_dir). A 4th tool invites the same malformation and competes for attention; a DONE: prefix disambiguates the existing text channel at zero schema cost. This is opencode's "require an explicit marker, don't trust the implicit stop", applied to the text channel.

Changes in _run_native (bridge.py):

1. NATIVE_SYSTEM (line 111–113): replace "reply with a summary and DO NOT call a tool" with → "When and ONLY when the task is fully done, reply with one line starting DONE: and a one-sentence result. Otherwise you MUST call a tool."
2. Track loop state: did_action (any tool ran), last_rc (parse the exit= already formatted by _run_shell_in_pty), nudges=0.
3. Replace the bare break (739–741) with a verdict gate:
   • text starts with DONE: → done (fast path, no nudge)
   • last_rc ∉ {0, None} and no DONE: → stalled (unresolved error)
   • no did_action → stalled (pure talk)
   • filler regex (let's proceed, next step, i will, trailing colon) and no DONE: → stalled
   • else (substantive text, action happened, no error) → accept as done — the single heuristic, on the SAFE side, so finished tasks aren't nudged every time (protects CPU latency, the cost a pure-structural form would incur here).
4. On stalled: append an output-aware, Goose-style nudge and continue, bounded by MAX_NUDGES = 2 (separate from the productive max_turns so real multi-step work isn't starved):

   [if last_rc≠0:] "The last command exited {rc} (failure) — fix that first (e.g. chmod +x before running a script)." + "You haven't signalled completion. If {task} is fully done reply DONE: …; otherwise call the next tool now — act, don't describe."

5. Wind-down: on the final turn, inject "reply DONE: with your result now" (opencode's MAX_STEPS pattern).
6. Keep _extract_text_tool_calls (already present) — opencode's "derive tool calls from parsed parts, not finish_reason" lesson, which this model needs.

Fixes mapped to observed failures: 3B early-stall → nudged to continue; give-up on error (both sizes) → nudged with the exit-code-aware message; over-nudging finished tasks → avoided by the DONE: fast path + safe-side accept.

Live validation (qwen2.5:3b, post-implementation)

Ran the two prior failure cases against the freshly-built loop; ground truth via podman exec hack-house.

• Multi-step (mkdir proj3 → write notes.txt → list): PASS — fixed. Previously stalled after step 1; now chained write_file → ls -1 proj3 to completion. Ground truth: proj3/notes.txt exists, contents hello world. The 3B's dominant failure mode is resolved by the loop alone (no model upgrade needed).
• Nudge fires on non-zero exit: confirmed. On the greet.sh case the PTY mirror showed ▸ (nudge: finish the task or reply DONE:) after a 126, i.e. the output-aware re-prompt triggered structurally as designed.
• Give-up-on-error is model-bound, not loop-bound. The 3B still can't recover the greet.sh task even when nudged: across runs it wrote self-referential content (echo "Hello, world!" > greet.sh), its chmod +x didn't stick (file left 644), and in one run it leaked a bare write_file proj3/greet.sh … tool call as summary text (not the <tool_call> JSON form _extract_text_tool_calls handles). These are 3B capability limits, consistent with the 3B-vs-7B table — the loop's job is to detect and report them honestly, not to make a weak model competent.

Honesty hardening added after live test

The weak 3B routinely claims success it didn't achieve ("written, made executable, and run successfully" after a 126; "Created greet.sh… ran it" with nothing on disk). Echoing that prose as the final summary is actively misleading, so the nudge-budget exhaustion path no longer trusts it blindly:

• never ran a tool → [stopped — model described the task but never ran a tool]
• left a non-zero exit → [stopped — last command exited {rc}; task likely incomplete] …
• clean, action-backed turn → the model's summary (unchanged).

Offline unit coverage: 23 assertions on _completion_verdict / _parse_exit / _strip_done / _nudge_message, all pass.

Benchmark suite + first baseline (bench/)

To compare harness/model changes systematically instead of eyeballing one-off runs, added a ground-truth-graded benchmark: bench/tasks.py (a 4-category × easy/medium/hard matrix — shell, code, git, multi) and bench/run.py (drives the live TUI via tmux, grades each task by a podman exec verify snippet, exit 0 == PASS). Tasks run in the agent's real cwd (/root) with bare filenames; pinning an absolute path instead just measured the weak model's absolute-path discipline and drowned out every other signal.

Runner gotcha (cost real time): the TUI is a full-screen (alt-screen) app, so tmux capture-pane -S does NOT yield chat history — only the current viewport. The first detector diffed a @user reply count and hung once old replies scrolled out of view. Fixed by keying completion off the is thinking… footer (engage → sustained-absence), which is viewport-independent.

Prompt-fairness fix (found via the benchmark): "…in your home directory" made literal-minded models create a home/ subdir (/root/home/a/b/c/marker.txt) or use ~/who.txt; dropped the phrase — bare filenames land in the agent's cwd (/root).

Baselines — four local CPU models (all tool-capable; fixed prompts)

Every candidate was probed first: qwen2.5(-coder) at 0.5b/1.5b/3b/7b all accept Ollama's tools field; deepseek-r1 and westenfelder/NL2SH reject it (HTTP 400 → degrade to the simple harness, useless for the native loop), llava is vision-only. One run each, shared room, CPU-only box:

model	size	PASS	avg / median s	passed
qwen2.5:0.5b	397 MB	1/12	55 / 49	shell-hard
qwen2.5-coder:1.5b	986 MB	1/12	67 / 53	shell-medium
qwen2.5:3b	1.9 GB	1/12	51 / 46	code-easy
qwen2.5-coder:7b	4.7 GB	2/12	141 / 106	shell-easy, multi-easy

Takeaways. All four cluster at 1–2/12 with high variance (which task passes flips run-to-run) — none reliably drives multi-step sandbox tasks in a shared room. The 7B doubles nothing: same pass band at ~3× the latency, so it is not worth it on this CPU box. qwen2.5:3b is the sweet spot (best speed, no worse quality) and stays the default. The low absolute scores are inflated downward by self-contamination (each task's NATIVE_CONTEXT=4 window pulls the previous task's command/summary) — a real deployment effect, but it means the suite's job is relative regression detection, not an absolute capability grade.

Dominant failure modes the summaries exposed, and where they point next:

failure mode	example	harness-addressable?
bare tool-call-as-text	write_file ./who.txt, mkdir proj, git clone … (esp. 0.5b — nearly every turn)	yes — extend _extract_text_tool_calls beyond <tool_call> JSON
<native> / <tools> tag leak in summary	<native> Cloned repository…, <tools>	yes — strip the tag
path hallucination	/home/qwen253b/conf_count.txt, ~/ unexpanded	partly (model)
content fabrication	wrote literal dell instead of running whoami	no (model capability)

The first row was hypothesised as the top priority — a pure harness parse gap. The benchmark now exists to prove whether fixing it moves the number.

Parser fix — what the benchmark actually proved (2026-06-10)

Generalised OllamaProvider._extract_text_tool_calls to recover a tool-call JSON object regardless of wrapper — qwen's <tool_call> tags, bare JSON, ```json fences, alternate tags (<tools>, <function_call>), OpenAI {"function":{…}} nesting, and parameters-vs-arguments. Gated on the known tool-name set (derived from the tools schema) so a stray JSON blob in prose — or a hallucinated make_dir — can never be coerced into an action. 11-case unit check: 8 leak shapes recover, 3 negatives (prose / unknown tool / random config JSON) ignored.

Result: it does NOT move the weak-model pass rate. Three 3b passes went 2 / 1 / 0 of 12 (prior baseline 1/12 — pure noise); two 0.5b passes went 0 / 0 (prior 1/12).

A direct /api/chat probe of 0.5b shows why — the hypothesis was wrong about the form of the leak. The weak models do not emit a JSON tool-call as text. They emit either:

• malformed structured tool_calls — e.g. write_file with content: null (the field IS populated; the arguments are just wrong → model capability, not a parse gap); or
• a fenced shell/code block in prose with NO tool call at all — e.g. content "…let's proceed:\n```bash\nmkdir -p a/b/c\n```", tool_calls: None. This is the "[stopped — model described the task but never ran a tool]" case that dominates the 0.5b table.

So the structured-JSON recovery is a correct, safe hardening (it WILL catch a real JSON-in-text leak from any model that produces one, and cannot fabricate an action), but the failure it was meant to fix is not the failure these CPU models actually have.

The real harness-addressable lever the data now points to: recover fenced code blocks (bash … / python … ) as run_shell calls when the turn carried no structured call. That is the genuine form of the weak-model leak. It is a bigger safety call than JSON recovery (it executes a block the model wrote as prose, which may be illustrative rather than an action), so it is left as an explicit decision, not folded in silently.

failure mode	example	harness-addressable?
malformed structured args	write_file with content:null	no (model capability)
fenced code in prose, no call	```bash\nmkdir -p a/b/c\n```, tool_calls:None	yes, but a safety call — parse fences → run_shell
JSON tool-call leaked as text	<tools>{"name":…}	yes — now handled (no weak-model effect)
path hallucination	/home/qwen253b/…, /ai/a/b/c, ~/ unexpanded	partly (model)
content fabrication	wrote literal octocat/dell instead of running whoami	no (model capability)

Other models worth pulling for a non-qwen data point (different family → different failure modes, both with strong native tool-calling): llama3.2:3b (~2 GB, peer to the 3B) and llama3.1:8b (~4.7 GB, peer to the 7B but general-purpose). Probe the tools field first, then bench/run.py --model <name>.

The first two are the clear next harness improvements; the benchmark now exists to prove whether they move the number.