docs(ai): 2026-06-10 native-harness findings + nudge-loop design

Live 3B-vs-7B command-entry results, observed failure modes (early stall,
give-up-on-error, tool hallucination), Goose/opencode loop-termination
research, and the output-aware dynamic-nudge-loop design to implement.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

docs/findings-native-harness-2026-06-10.md

@@ -0,0 +1,127 @@
+# Native harness — live command-entry findings + nudge-loop design
+
+**Date:** 2026-06-10
+**Context:** Evaluating the `native` tool-calling harness (`cmd_chat/agent/bridge.py`,
+`_run_native`) on its ability to drive shell commands into the shared podman/Kali
+sandbox, on a CPU-only box (no GPU). Two prior fixes were live-validated here and
+committed in `a0aa14d`: §3 PTY-sentinel (`_run_shell_in_pty`) and `NATIVE_CONTEXT=4`.
+
+## Test setup
+
+- Fresh room, podman sandbox (`hack-house` container), `/grant ai` drive.
+- Models compared: `qwen2.5:3b` (1.9 GB, prior default) and `qwen2.5-coder:7b`
+  (4.7 GB, pulled for this test). `deepseek-r1:latest` and `westenfelder/NL2SH`
+  were ruled out — both **reject Ollama's `tools` field** (`HTTP 400 "does not
+  support tools"`), so they degrade to the simple injector and can't exercise the
+  native loop.
+- Each task verified against **ground truth** (podman exec into the container),
+  not just the chat/PTY surface.
+
+## Results: qwen2.5:3b vs qwen2.5-coder:7b
+
+| Behaviour | qwen2.5:3b | qwen2.5-coder:7b |
+|---|---|---|
+| Single command (`whoami`) | PASS | PASS |
+| write+run (`hello.sh` date+hostname) | PASS, accurate summary | PASS |
+| **Multi-step** (mkdir→write→list) | **FAIL** — stalled after step 1 ("ok, let's proceed to the next step") | PASS — chained make_dir→write→read, completed |
+| **Recover from mid-task error** | FAIL — gives up | INCONSISTENT — recovered from `[unknown tool make_dir]`, but on `greet.sh` skipped chmod, hit exit 126, then **stopped** |
+| Tool-schema discipline | clean | hallucinated a `make_dir` tool (not in schema) — wasted a turn (`[unknown tool make_dir]`, bridge.py:597) |
+| Final summary quality | vague ("I will proceed…") | empty → `(done)` fallback, or raw advice text |
+| Latency (CPU, no GPU) | ~20–40 s/task | ~2–4× slower, 30–90 s/turn |
+
+### Failure modes observed (both sizes)
+
+1. **Early termination on multi-step** (3B dominant): the model emits filler text
+   with no tool call mid-task; the loop reads "text + no tool call = done" and stops.
+   The remaining steps never run.
+2. **Give-up on unresolved non-zero exit** (both sizes): after a failing command
+   (exit 126/127), the model emits a summary/advice instead of fixing the cause.
+   greet.sh left at mode 644 (never chmod'd) despite the task saying "make it
+   executable".
+3. **Tool hallucination** (7B): invented `make_dir`; the schema only has
+   run_shell/write_file/read_file.
+
+### Root cause in code
+
+`_run_native`, bridge.py:739–741:
+```python
+if not calls:
+    final = (text or "").strip()
+    break        # ANY text-without-toolcall ends the task
+```
+NATIVE_SYSTEM (line 111–113) *also* tells the model to signal completion this exact
+way. So one signal — "text, no tool call" — is overloaded to mean BOTH "done" and
+"stalling/thinking". Disambiguating it is the fix.
+
+**Takeaway:** a bigger model buys multi-step persistence (the 3B's main flaw) but
+does NOT eliminate the give-up-on-error mode, and costs heavy latency. A nudge loop
+that keys off **exit codes** (not just filler text) earns its keep at both sizes.
+
+## Research: how Goose & opencode structure loop termination
+
+(Source read directly from clones; key files cited.)
+
+- **Goose** (`crates/goose/src/agents/agent.rs`): tracks `no_tools_called`
+  (line 1801). When true it does NOT just stop (lines 2232–2316) — in
+  structured/goal modes it injects a continuation nudge
+  (`FINAL_OUTPUT_CONTINUATION_MESSAGE` = "You MUST call the final_output tool
+  NOW…") and loops. Completion is an **explicit terminal tool**
+  (`recipe__final_output`, `final_output_tool.rs`). Vanilla chat with no
+  recipe/goal does fall through to text-as-done, but every structured path
+  overrides that. Bounded by `max_turns`.
+- **opencode** (`packages/opencode/src/session/prompt.ts:1156–1183`): exits only
+  when finish is a real stop reason **AND** it independently re-derives that there
+  are no pending tool calls from the parsed message parts — comment: *"Some
+  providers return 'stop' even when the assistant message contains tool calls."*
+  Hard step cap (`maxSteps`) with a wind-down message (`MAX_STEPS`,
+  `prompt/max-steps.txt`); structured mode forces `toolChoice: "required"`.
+- **Canonical / Cline-Roo**: weak-model harnesses favour an explicit completion
+  action (`attempt_completion`/`submit`) over trusting "empty tool_calls = done".
+- **Recommendation from research:** go structural, not filler-heuristic ("a losing
+  arms race"); nudge on text-without-completion; a hard cap is the only
+  unconditional exit; derive "did a tool get called" from parsed parts, not
+  `finish_reason` (qwen on CPU is unreliable there — it leaks tool calls as
+  `<tool_call>` text in `content`; already handled by
+  `OllamaProvider._extract_text_tool_calls`).
+
+## Design: dynamic nudge loop (output-aware, structural terminator)
+
+Structural termination via a **`DONE:` text sentinel** rather than a 4th tool.
+Divergence from the research's "add a done tool", justified for THIS model:
+qwen-on-CPU already mis-emits structured tool calls (leaks as `<tool_call>` text;
+the 7B even hallucinated `make_dir`). A 4th tool invites the same malformation and
+competes for attention; a `DONE:` prefix disambiguates the existing text channel at
+zero schema cost. This is opencode's "require an explicit marker, don't trust the
+implicit stop", applied to the text channel.
+
+Changes in `_run_native` (bridge.py):
+
+1. **NATIVE_SYSTEM** (line 111–113): replace "reply with a summary and DO NOT call
+   a tool" with → *"When and ONLY when the task is fully done, reply with one line
+   starting `DONE:` and a one-sentence result. Otherwise you MUST call a tool."*
+2. **Track loop state:** `did_action` (any tool ran), `last_rc` (parse the `exit=`
+   already formatted by `_run_shell_in_pty`), `nudges=0`.
+3. **Replace the bare `break` (739–741) with a verdict gate:**
+   - text starts with `DONE:` → **done** (fast path, no nudge)
+   - `last_rc` ∉ {0, None} and no `DONE:` → **stalled** (unresolved error)
+   - no `did_action` → **stalled** (pure talk)
+   - filler regex (`let's proceed`, `next step`, `i will`, trailing colon) and no
+     `DONE:` → **stalled**
+   - else (substantive text, action happened, no error) → **accept as done** — the
+     single heuristic, on the SAFE side, so finished tasks aren't nudged every time
+     (protects CPU latency, the cost a pure-structural form would incur here).
+4. **On `stalled`:** append an output-aware, Goose-style nudge and `continue`,
+   bounded by `MAX_NUDGES = 2` (separate from the productive `max_turns` so real
+   multi-step work isn't starved):
+   > *[if last_rc≠0:]* "The last command exited {rc} (failure) — fix that first
+   > (e.g. `chmod +x` before running a script)." + "You haven't signalled
+   > completion. If `{task}` is fully done reply `DONE: …`; otherwise call the next
+   > tool now — act, don't describe."
+5. **Wind-down:** on the final turn, inject "reply `DONE:` with your result now"
+   (opencode's `MAX_STEPS` pattern).
+6. **Keep** `_extract_text_tool_calls` (already present) — opencode's "derive tool
+   calls from parsed parts, not finish_reason" lesson, which this model needs.
+
+Fixes mapped to observed failures: 3B early-stall → nudged to continue; give-up on
+error (both sizes) → nudged with the exit-code-aware message; over-nudging finished
+tasks → avoided by the `DONE:` fast path + safe-side accept.