Trilltechnician/hack-house
1
3
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

test(ai): ground-truth benchmark for the native harness + first baseline

Browse Source 
Add bench/ — a 4-category × easy/medium/hard task matrix (shell, code, git,
multi) and a runner that drives the live TUI over tmux and grades each task by
a `podman exec` verify snippet (exit 0 == PASS), never by the model's
self-reported summary (which the weak CPU model fabricates). Tasks run in the
agent's real cwd with bare filenames so the suite measures task completion, not
the model's absolute-path discipline. Completion is detected off the viewport-
independent `is thinking…` footer (the TUI is full-screen, so capture-pane
scrollback is not chat history).

First baseline (qwen2.5:3b): 2/12 PASS, high variance. Surfaces the next
harness-addressable improvements — `<native>` tag leakage and bare
tool-call-as-text — now measurable against this suite. Findings doc updated.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
leetcrypt
2026-06-10 01:30:09 -07:00
parent 156e9fe176
commit df45303f5c
6 changed files with 598 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/findings-native-harness-2026-06-10.md
+33
View File
@@ -161,3 +161,36 @@ exhaustion path no longer trusts it blindly:
Offline unit coverage: 23 assertions on `_completion_verdict` / `_parse_exit` /
`_strip_done` / `_nudge_message`, all pass.
## Benchmark suite + first baseline (`bench/`)
To compare harness/model changes **systematically** instead of eyeballing one-off
runs, added a ground-truth-graded benchmark: `bench/tasks.py` (a 4-category ×
easy/medium/hard matrix — shell, code, git, multi) and `bench/run.py` (drives the
live TUI via tmux, grades each task by a `podman exec` verify snippet, exit 0 ==
PASS). Tasks run in the agent's real cwd (`/root`) with bare filenames; pinning an
absolute path instead just measured the weak model's absolute-path discipline and
drowned out every other signal.
**Runner gotcha (cost real time):** the TUI is a full-screen (alt-screen) app, so
`tmux capture-pane -S` does NOT yield chat history — only the current viewport.
The first detector diffed a `@user` reply count and hung once old replies scrolled
out of view. Fixed by keying completion off the **`is thinking…` footer** (engage →
sustained-absence), which is viewport-independent.
**Baseline — qwen2.5:3b, 2/12 PASS** (`bench/results/baseline-qwen2.5_3b.json`).
Pass: shell-medium (nested mkdir+write), multi-easy (mkdir→write→list). The run is
**high-variance** — across two runs git-easy and shell-medium each flipped PASS↔FAIL
— so treat a single run as a smoke signal, not a precise score; average several (or
move to the 7B) for a stable number. Dominant failure modes the summaries exposed,
and where they point next:
| failure mode | example | harness-addressable? |
|---|---|---|
| `<native>` tag leak in summary | `<native> Cloned repository…` | yes — strip the tag |
| bare tool-call-as-text | `write_file ./wc.py`, `writeFile "…"` | yes — extend `_extract_text_tool_calls` beyond `<tool_call>` JSON |
| path hallucination | `/home/qwen253b/conf_count.txt`, `~/` unexpanded | partly (model) |
| content fabrication | wrote literal `dell` instead of running whoami | no (model capability) |
The first two are the clear next harness improvements; the benchmark now exists to
prove whether they move the number.