feat(ai): output-aware nudge loop with DONE: terminator for native harness

Replace the overloaded "text + no tool call = done" terminator that made the
weak CPU model stall mid-task or give up after a failing command. Termination
is now a structural `DONE:` text sentinel; a text-only turn is resolved by an
output-aware verdict (DONE: marker / unresolved non-zero exit / no action /
filler language) and re-prompted with an exit-code-aware nudge, bounded by
MAX_NUDGES on top of max_turns. On exhaustion the summary is honest rather than
echoing the model's false "run successfully" — it reports when no tool ran or a
command exited non-zero. Live-validated on qwen2.5:3b: the multi-step stall is
fixed (proj3 completes end-to-end, ground-truth confirmed); the nudge fires on
a 126; residual give-up is model-bound. 23 offline unit assertions pass.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
leetcrypt
2026-06-10 00:31:19 -07:00
parent d5f423024e
commit 156e9fe176
2 changed files with 147 additions and 7 deletions
@@ -125,3 +125,39 @@ Changes in `_run_native` (bridge.py):
Fixes mapped to observed failures: 3B early-stall → nudged to continue; give-up on
error (both sizes) → nudged with the exit-code-aware message; over-nudging finished
tasks → avoided by the `DONE:` fast path + safe-side accept.
## Live validation (qwen2.5:3b, post-implementation)
Ran the two prior failure cases against the freshly-built loop; ground truth via
`podman exec hack-house`.
- **Multi-step (`mkdir proj3 → write notes.txt → list`): PASS — fixed.** Previously
stalled after step 1; now chained write_file → `ls -1 proj3` to completion.
Ground truth: `proj3/notes.txt` exists, contents `hello world`. The 3B's dominant
failure mode is resolved by the loop alone (no model upgrade needed).
- **Nudge fires on non-zero exit: confirmed.** On the `greet.sh` case the PTY mirror
showed `▸ (nudge: finish the task or reply DONE:)` after a 126, i.e. the
output-aware re-prompt triggered structurally as designed.
- **Give-up-on-error is model-bound, not loop-bound.** The 3B still can't *recover*
the greet.sh task even when nudged: across runs it wrote self-referential content
(`echo "Hello, world!" > greet.sh`), its `chmod +x` didn't stick (file left 644),
and in one run it leaked a bare `write_file proj3/greet.sh …` tool call **as
summary text** (not the `<tool_call>` JSON form `_extract_text_tool_calls`
handles). These are 3B capability limits, consistent with the 3B-vs-7B table —
the loop's job is to detect and report them honestly, not to make a weak model
competent.
### Honesty hardening added after live test
The weak 3B routinely *claims* success it didn't achieve ("written, made executable,
and run successfully" after a 126; "Created greet.sh… ran it" with nothing on disk).
Echoing that prose as the final summary is actively misleading, so the nudge-budget
exhaustion path no longer trusts it blindly:
- never ran a tool → `[stopped — model described the task but never ran a tool]`
- left a non-zero exit → `[stopped — last command exited {rc}; task likely
incomplete] …`
- clean, action-backed turn → the model's summary (unchanged).
Offline unit coverage: 23 assertions on `_completion_verdict` / `_parse_exit` /
`_strip_done` / `_nudge_message`, all pass.