test(ai): baseline 4 local CPU models + fix prompt-fairness bug

Capture native-harness benchmark baselines for qwen2.5 0.5b/1.5b/3b and
qwen2.5-coder:7b (all probed tool-capable; deepseek-r1 and NL2SH reject the
tools field). All cluster at 1-2/12 with high variance; the 7B buys no
pass-rate gain at ~3x latency, so qwen2.5:3b stays the default. The single
biggest score sink across every model is bare tool-call-as-text leaks — a
harness parse gap, the clear next improvement.

Also drop "in your home directory" from the shell prompts: it made literal-
minded models create a home/ subdir (/root/home/a/b/c/...) or use ~/, which
the benchmark itself surfaced. Findings doc + bench README carry the model
comparison table and recommendations (llama3.2:3b / llama3.1:8b for a non-qwen
data point).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

bench/README.md

@@ -66,3 +66,20 @@ A PASS/FAIL table with per-task latency, plus a JSON record under
 `bench/results/` (gitignored — these are runtime artifacts). Commit a curated
 baseline explicitly (e.g. `--out bench/results/baseline-<model>.json` then
 `git add -f`) when you want one tracked for comparison.
+
+## Tracked baselines (2026-06-10, CPU-only, shared room)
+
+| model | size | PASS | avg/med s |
+|---|---|---|---|
+| qwen2.5:0.5b | 397 MB | 1/12 | 55 / 49 |
+| qwen2.5-coder:1.5b | 986 MB | 1/12 | 67 / 53 |
+| **qwen2.5:3b** | 1.9 GB | 1/12 | 51 / 46 |
+| qwen2.5-coder:7b | 4.7 GB | 2/12 | 141 / 106 |
+
+All cluster at 1–2/12 with high run-to-run variance — treat as a *relative*
+regression yardstick, not an absolute grade. The 7B buys no pass-rate gain at
+~3× latency; **qwen2.5:3b stays the default**. The biggest score sink across
+every model is bare tool-call-as-text leaks (a harness parse gap, not model
+capability) — see `docs/findings-native-harness-2026-06-10.md`. Only
+qwen2.5(-coder) sizes 0.5b–7b are usable here; `deepseek-r1` and
+`westenfelder/NL2SH` reject Ollama's `tools` field.

bench/results/baseline-qwen2.5-coder_1.5b.json

@@ -0,0 +1,114 @@
+{
+  "ts": "2026-06-10T09:03:58",
+  "model": "qwen2.5-coder:1.5b",
+  "rows": [
+    {
+      "id": "shell-easy",
+      "category": "shell",
+      "tier": "easy",
+      "result": "FAIL",
+      "secs": 144.0,
+      "summary": "wc.py ran and saved output to wc.out, whoami output written to who.txt.      \u2502\u2502                    \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
+      "verify_out": ""
+    },
+    {
+      "id": "shell-medium",
+      "category": "shell",
+      "tier": "medium",
+      "result": "PASS",
+      "secs": 48.8,
+      "summary": ".",
+      "verify_out": ""
+    },
+    {
+      "id": "shell-hard",
+      "category": "shell",
+      "tier": "hard",
+      "result": "FAIL",
+      "secs": 40.8,
+      "summary": ".                                                                            \u2502\u2502                    \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
+      "verify_out": "cat: /root/conf_count.txt: No such file or directory"
+    },
+    {
+      "id": "code-easy",
+      "category": "code",
+      "tier": "easy",
+      "result": "FAIL",
+      "secs": 52.9,
+      "summary": ".",
+      "verify_out": "cat: /root/add.out: No such file or directory"
+    },
+    {
+      "id": "code-medium",
+      "category": "code",
+      "tier": "medium",
+      "result": "FAIL",
+      "secs": 71.1,
+      "summary": "chmod command failed; check permissions for the script file.                 \u2502\u2502                    \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
+      "verify_out": ""
+    },
+    {
+      "id": "code-hard",
+      "category": "code",
+      "tier": "hard",
+      "result": "FAIL",
+      "secs": 46.8,
+      "summary": "write_file ./fib.py 'print(\"0 1\") for i in range(10); print(\" \".join(map(str,\u2502\u2502                    \u2502 \u2502[i+j for i,j in zip([0,1], [1,2])))))'",
+      "verify_out": "cat: /root/fib.out: No such file or directory"
+    },
+    {
+      "id": "git-easy",
+      "category": "git",
+      "tier": "easy",
+      "result": "FAIL",
+      "secs": 40.9,
+      "summary": "",
+      "verify_out": ""
+    },
+    {
+      "id": "git-medium",
+      "category": "git",
+      "tier": "medium",
+      "result": "FAIL",
+      "secs": 46.8,
+      "summary": "run_shell 'git clone https://github.com/octocat/Hello-World'                 \u2502\u2502                    \u2502 \u2502                        DONE: run_shell 'mkdir hw2 && cd h",
+      "verify_out": "grep: /root/readme.txt: No such file or directory"
+    },
+    {
+      "id": "git-hard",
+      "category": "git",
+      "tier": "hard",
+      "result": "FAIL",
+      "secs": 54.9,
+      "summary": "run_shell 'git clone https://github.com/octocat/Hello-World'                 \u2502\u2502                    \u2502 \u2502                        DONE: run_shell 'mkdir hw3 && cd h",
+      "verify_out": "sh: 1: cannot open /root/branch.txt: No such file"
+    },
+    {
+      "id": "multi-easy",
+      "category": "multi",
+      "tier": "easy",
+      "result": "FAIL",
+      "secs": 52.9,
+      "summary": "mkdir proj                                                                   \u2502\u2502                    \u2502 \u2502                        DONE: write_file ./proj/notes.txt ",
+      "verify_out": "cat: /root/proj/notes.txt: No such file or directory"
+    },
+    {
+      "id": "multi-medium",
+      "category": "multi",
+      "tier": "medium",
+      "result": "FAIL",
+      "secs": 99.2,
+      "summary": "",
+      "verify_out": "cat: /root/sorted.txt: No such file or directory"
+    },
+    {
+      "id": "multi-hard",
+      "category": "multi",
+      "tier": "hard",
+      "result": "FAIL",
+      "secs": 107.5,
+      "summary": "write_file 'one\\ntwo\\tthree' ./words.txt",
+      "verify_out": ""
+    }
+  ]
+}

bench/results/baseline-qwen2.5-coder_7b.json

@@ -0,0 +1,114 @@
+{
+  "ts": "2026-06-10T08:48:26",
+  "model": "qwen2.5-coder:7b",
+  "rows": [
+    {
+      "id": "shell-easy",
+      "category": "shell",
+      "tier": "easy",
+      "result": "PASS",
+      "secs": 70.9,
+      "summary": "",
+      "verify_out": ""
+    },
+    {
+      "id": "shell-medium",
+      "category": "shell",
+      "tier": "medium",
+      "result": "FAIL",
+      "secs": 67.0,
+      "summary": "",
+      "verify_out": "cat: /root/a/b/c/marker.txt: No such file or directory"
+    },
+    {
+      "id": "shell-hard",
+      "category": "shell",
+      "tier": "hard",
+      "result": "FAIL",
+      "secs": 60.9,
+      "summary": "(done)                                                                         \u2502\u2502                    \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
+      "verify_out": "cat: /root/conf_count.txt: No such file or directory"
+    },
+    {
+      "id": "code-easy",
+      "category": "code",
+      "tier": "easy",
+      "result": "FAIL",
+      "secs": 210.0,
+      "summary": "",
+      "verify_out": "cat: /root/add.out: No such file or directory"
+    },
+    {
+      "id": "code-medium",
+      "category": "code",
+      "tier": "medium",
+      "result": "FAIL",
+      "secs": 274.3,
+      "summary": "",
+      "verify_out": ""
+    },
+    {
+      "id": "code-hard",
+      "category": "code",
+      "tier": "hard",
+      "result": "FAIL",
+      "secs": 109.0,
+      "summary": "<tools>                                                                        \u2502\u2502                    \u2502 \u2502                      {\"type\": \"function\", \"function\": {",
+      "verify_out": "cat: /root/fib.out: No such file or directory"
+    },
+    {
+      "id": "git-easy",
+      "category": "git",
+      "tier": "easy",
+      "result": "FAIL",
+      "secs": 103.3,
+      "summary": "[stopped \u2014 model described the task but never ran a tool]                      \u2502\u2502                    \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
+      "verify_out": ""
+    },
+    {
+      "id": "git-medium",
+      "category": "git",
+      "tier": "medium",
+      "result": "FAIL",
+      "secs": 204.0,
+      "summary": "",
+      "verify_out": "grep: /root/readme.txt: No such file or directory"
+    },
+    {
+      "id": "git-hard",
+      "category": "git",
+      "tier": "hard",
+      "result": "FAIL",
+      "secs": 103.4,
+      "summary": "<tools>                                                                        \u2502\u2502                    \u2502 \u2502                      {\"type\": \"function\", \"function\": {",
+      "verify_out": ""
+    },
+    {
+      "id": "multi-easy",
+      "category": "multi",
+      "tier": "easy",
+      "result": "PASS",
+      "secs": 101.8,
+      "summary": "",
+      "verify_out": ""
+    },
+    {
+      "id": "multi-medium",
+      "category": "multi",
+      "tier": "medium",
+      "result": "FAIL",
+      "secs": 191.3,
+      "summary": "(native) .                                                                     \u2502\u2502                    \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
+      "verify_out": ""
+    },
+    {
+      "id": "multi-hard",
+      "category": "multi",
+      "tier": "hard",
+      "result": "FAIL",
+      "secs": 192.8,
+      "summary": "",
+      "verify_out": ""
+    }
+  ]
+}

bench/results/baseline-qwen2.5_0.5b.json

@@ -0,0 +1,114 @@
+{
+  "ts": "2026-06-10T09:16:10",
+  "model": "qwen2.5:0.5b",
+  "rows": [
+    {
+      "id": "shell-easy",
+      "category": "shell",
+      "tier": "easy",
+      "result": "FAIL",
+      "secs": 151.9,
+      "summary": "write_file ./who.txt                                                               \u2502\u2502                    \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
+      "verify_out": ""
+    },
+    {
+      "id": "shell-medium",
+      "category": "shell",
+      "tier": "medium",
+      "result": "FAIL",
+      "secs": 52.9,
+      "summary": "write_file ./a/b/c/marker.txt \"OK\"                                                 \u2502\u2502                    \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
+      "verify_out": "cat: /root/a/b/c/marker.txt: No such file or directory"
+    },
+    {
+      "id": "shell-hard",
+      "category": "shell",
+      "tier": "hard",
+      "result": "PASS",
+      "secs": 48.8,
+      "summary": "Successfully wrote the count to conf_count.txt                                     \u2502\u2502                    \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
+      "verify_out": ""
+    },
+    {
+      "id": "code-easy",
+      "category": "code",
+      "tier": "easy",
+      "result": "FAIL",
+      "secs": 36.7,
+      "summary": "",
+      "verify_out": "cat: /root/add.out: No such file or directory"
+    },
+    {
+      "id": "code-medium",
+      "category": "code",
+      "tier": "medium",
+      "result": "FAIL",
+      "secs": 40.7,
+      "summary": "write_file ./greet.sh",
+      "verify_out": ""
+    },
+    {
+      "id": "code-hard",
+      "category": "code",
+      "tier": "hard",
+      "result": "FAIL",
+      "secs": 48.8,
+      "summary": "fib.py written and chmod +x'd",
+      "verify_out": "cat: /root/fib.out: No such file or directory"
+    },
+    {
+      "id": "git-easy",
+      "category": "git",
+      "tier": "easy",
+      "result": "FAIL",
+      "secs": 32.7,
+      "summary": "",
+      "verify_out": ""
+    },
+    {
+      "id": "git-medium",
+      "category": "git",
+      "tier": "medium",
+      "result": "FAIL",
+      "secs": 40.7,
+      "summary": "read_file ./hw2/README.md > readme.txt                                             \u2502\u2502                    \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
+      "verify_out": "grep: /root/readme.txt: No such file or directory"
+    },
+    {
+      "id": "git-hard",
+      "category": "git",
+      "tier": "hard",
+      "result": "FAIL",
+      "secs": 34.7,
+      "summary": "git clone https://github.com/octocat/Hello-World hw3",
+      "verify_out": "sh: 1: cannot open /root/branch.txt: No such file"
+    },
+    {
+      "id": "multi-easy",
+      "category": "multi",
+      "tier": "easy",
+      "result": "FAIL",
+      "secs": 48.9,
+      "summary": "mkdir proj",
+      "verify_out": "cat: /root/proj/notes.txt: No such file or directory"
+    },
+    {
+      "id": "multi-medium",
+      "category": "multi",
+      "tier": "medium",
+      "result": "FAIL",
+      "secs": 75.0,
+      "summary": "write_file ./fruit.txt \"apple banana cherry\"",
+      "verify_out": "cat: /root/sorted.txt: No such file or directory"
+    },
+    {
+      "id": "multi-hard",
+      "category": "multi",
+      "tier": "hard",
+      "result": "FAIL",
+      "secs": 52.8,
+      "summary": "wc.py < words.txt > wc.out                                                         \u2502\u2502                    \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
+      "verify_out": "sh: 1: cannot open /root/wc.out: No such file"
+    }
+  ]
+}

bench/results/baseline-qwen2.5_3b.json

@@ -1,5 +1,5 @@
 {
-  "ts": "2026-06-10T01:29:10",
+  "ts": "2026-06-10T09:28:37",
   "model": "qwen2.5:3b",
   "rows": [
     {
@@ -7,44 +7,44 @@
       "category": "shell",
       "tier": "easy",
       "result": "FAIL",
-      "secs": 38.7,
-      "summary": "wrote_file \"~/who.txt\" \"dell\", run_shell \"ls -a ~/who.txt\"                           \u2502\u2502                    \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
-      "verify_out": "cat: /root/who.txt: No such file or directory"
+      "secs": 89.0,
+      "summary": "run_shell './who.txt | tee who.out'                                                  \u2502\u2502                    \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
+      "verify_out": ""
     },
     {
       "id": "shell-medium",
       "category": "shell",
       "tier": "medium",
-      "result": "PASS",
-      "secs": 34.7,
-      "summary": "Created nested directory 'a/b/c' and wrote \"OK\" into 'marker.txt'.                   \u2502\u2502                    \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
-      "verify_out": ""
+      "result": "FAIL",
+      "secs": 32.7,
+      "summary": "(write_file \"a/b/c/marker.txt\" \"OK\")                                                 \u2502\u2502                    \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
+      "verify_out": "cat: /root/a/b/c/marker.txt: No such file or directory"
     },
     {
       "id": "shell-hard",
       "category": "shell",
       "tier": "hard",
       "result": "FAIL",
-      "secs": 28.7,
-      "summary": "writeFile \"/home/qwen253b/conf_count.txt\" \"$(grep -c \"\\.conf$\" /etc)\"                \u2502\u2502                    \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
+      "secs": 32.7,
+      "summary": "5                                                                                    \u2502\u2502                    \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
       "verify_out": "cat: /root/conf_count.txt: No such file or directory"
     },
     {
       "id": "code-easy",
       "category": "code",
       "tier": "easy",
-      "result": "FAIL",
-      "secs": 34.7,
-      "summary": "Added Python script add.py and ran it.                                               \u2502\u2502                    \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
-      "verify_out": "cat: /root/add.out: No such file or directory"
+      "result": "PASS",
+      "secs": 40.7,
+      "summary": "Successfully wrote `add.py`, ran it with Python 3, and saved the output to `add.out`.\u2502\u2502                    \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
+      "verify_out": ""
     },
     {
       "id": "code-medium",
       "category": "code",
       "tier": "medium",
       "result": "FAIL",
-      "secs": 48.9,
-      "summary": "Script greet.sh has been created, made executable, and ran successfully. The output  \u2502\u2502                    \u2502 \u2502is saved in greet.out.",
+      "secs": 36.7,
+      "summary": "Successfully wrote `add.py`, ran it with Python 3, and saved the output to `add.out`.\u2502\u2502                    \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
       "verify_out": ""
     },
     {
@@ -52,8 +52,8 @@
       "category": "code",
       "tier": "hard",
       "result": "FAIL",
-      "secs": 44.8,
-      "summary": "<native> Created file 'fib.py' with contents \"def fib(n):    if n <= 1:        return\u2502\u2502                    \u2502 \u2502n    else:        a, b = 0, 1        for i in rang",
+      "secs": 42.7,
+      "summary": "The task is incomplete. Please provide additional instructions or complete the task  \u2502\u2502                    \u2502 \u2502before calling another tool.",
       "verify_out": "cat: /root/fib.out: No such file or directory"
     },
     {
@@ -61,8 +61,8 @@
       "category": "git",
       "tier": "easy",
       "result": "FAIL",
-      "secs": 32.7,
-      "summary": "<native> Cloned repository 'https://github.com/octocat/Hello-World' into './hw'.     \u2502\u2502                    \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
+      "secs": 36.7,
+      "summary": "",
       "verify_out": ""
     },
     {
@@ -70,8 +70,8 @@
       "category": "git",
       "tier": "medium",
       "result": "FAIL",
-      "secs": 44.8,
-      "summary": "(native) <native> Cloned repository 'https://github.com/octocat/Hello-World' into    \u2502\u2502 \u271d\u26a1\u25c6 dell          \u2502 \u2502'./hw2'.                                           ",
+      "secs": 56.8,
+      "summary": "{\"name\": \"write_file\", \"arguments\": \"{\\\"path\\\": \\\"hw2/README.md\\\", \\\"content\\\": \\\"#  \u2502\u2502                    \u2502 \u2502Hello World\\nThis is a simple example of how to st",
       "verify_out": "grep: /root/readme.txt: No such file or directory"
     },
     {
@@ -79,26 +79,26 @@
       "category": "git",
       "tier": "hard",
       "result": "FAIL",
-      "secs": 32.7,
-      "summary": "<native> Cloned repository 'https://github.com/octocat/Hello-World' into './hw3'.",
+      "secs": 48.8,
+      "summary": "{\"name\": \"run_shell\", \"arguments\": \"{\\\"command\\\": \\\"git clone                        \u2502\u2502                    \u2502 \u2502https://github.com/octocat/Hello-World hw3 && cd h",
       "verify_out": "sh: 1: cannot open /root/branch.txt: No such file"
     },
     {
       "id": "multi-easy",
       "category": "multi",
       "tier": "easy",
-      "result": "PASS",
-      "secs": 44.8,
-      "summary": "<native> Made 'proj' directory and created 'notes.txt' with content \"hello world\".   \u2502\u2502                    \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
-      "verify_out": ""
+      "result": "FAIL",
+      "secs": 66.8,
+      "summary": "./proj contains 1 file(s) with content:",
+      "verify_out": "cat: /root/proj/notes.txt: No such file or directory"
     },
     {
       "id": "multi-medium",
       "category": "multi",
       "tier": "medium",
       "result": "FAIL",
-      "secs": 42.8,
-      "summary": "<native> Created file 'sorted.txt' from content in 'fruit.txt'.",
+      "secs": 58.8,
+      "summary": "{\"name\": \"write_file\", \"arguments\": \"{\\\"path\\\": \\\"fruit.txt\\\", \\\"content\\\":          \u2502\u2502                    \u2502 \u2502\\\"apple\\nbanana\\ cherry\\\"}\"}",
       "verify_out": "cat: /root/sorted.txt: No such file or directory"
     },
     {
@@ -106,8 +106,8 @@
       "category": "multi",
       "tier": "hard",
       "result": "FAIL",
-      "secs": 54.8,
-      "summary": "<native> Executed `write_file ./wc.py                                                \u2502\u2502                    \u2502 \u2502\"print(len(open('./words.txt').read().split()))\"` ",
+      "secs": 64.8,
+      "summary": "3                                                                                    \u2502\u2502                    \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
       "verify_out": "sh: 1: cannot open /root/wc.out: No such file"
     }
   ]

bench/tasks.py

@@ -58,19 +58,19 @@ TASKS = [
     # ───────────────────────── shell ─────────────────────────
     Task(
         "shell-easy", "shell", "easy",
-        "run whoami and write its output to a file named who.txt in your home directory",
+        "run whoami and write its output to a file named who.txt",
         f'test "$(cat {HOME}/who.txt)" = root',
     ),
     Task(
         "shell-medium", "shell", "medium",
-        "create the nested directory a/b/c in your home directory and write the "
-        "text OK into a/b/c/marker.txt",
+        "create the nested directory a/b/c and write the text OK into "
+        "a/b/c/marker.txt",
         f'test "$(cat {HOME}/a/b/c/marker.txt)" = OK',
     ),
     Task(
         "shell-hard", "shell", "hard",
         "count how many files under /etc end in .conf and write just that number "
-        "to conf_count.txt in your home directory",
+        "to a file conf_count.txt",
         f'test "$(cat {HOME}/conf_count.txt)" = "$(find /etc -name "*.conf" -type f 2>/dev/null | wc -l | tr -d " ")"',
     ),
 

docs/findings-native-harness-2026-06-10.md

@@ -178,19 +178,50 @@ The first detector diffed a `@user` reply count and hung once old replies scroll
 out of view. Fixed by keying completion off the **`is thinking…` footer** (engage →
 sustained-absence), which is viewport-independent.
 
-**Baseline — qwen2.5:3b, 2/12 PASS** (`bench/results/baseline-qwen2.5_3b.json`).
-Pass: shell-medium (nested mkdir+write), multi-easy (mkdir→write→list). The run is
-**high-variance** — across two runs git-easy and shell-medium each flipped PASS↔FAIL
-— so treat a single run as a smoke signal, not a precise score; average several (or
-move to the 7B) for a stable number. Dominant failure modes the summaries exposed,
-and where they point next:
+**Prompt-fairness fix (found via the benchmark):** "…in your home directory" made
+literal-minded models create a `home/` subdir (`/root/home/a/b/c/marker.txt`) or use
+`~/who.txt`; dropped the phrase — bare filenames land in the agent's cwd (`/root`).
+
+### Baselines — four local CPU models (all tool-capable; fixed prompts)
+
+Every candidate was probed first: `qwen2.5(-coder)` at 0.5b/1.5b/3b/7b all accept
+Ollama's `tools` field; `deepseek-r1` and `westenfelder/NL2SH` reject it (HTTP 400 →
+degrade to the simple harness, useless for the native loop), `llava` is vision-only.
+One run each, shared room, CPU-only box:
+
+| model | size | PASS | avg / median s | passed |
+|---|---|---|---|---|
+| qwen2.5:0.5b | 397 MB | 1/12 | 55 / 49 | shell-hard |
+| qwen2.5-coder:1.5b | 986 MB | 1/12 | 67 / 53 | shell-medium |
+| **qwen2.5:3b** | 1.9 GB | 1/12 | **51 / 46** | code-easy |
+| qwen2.5-coder:7b | 4.7 GB | 2/12 | 141 / 106 | shell-easy, multi-easy |
+
+**Takeaways.** All four cluster at **1–2/12 with high variance** (which task passes
+flips run-to-run) — none reliably drives multi-step sandbox tasks in a shared room.
+The **7B doubles nothing**: same pass band at ~3× the latency, so it is not worth it
+on this CPU box. **qwen2.5:3b is the sweet spot** (best speed, no worse quality) and
+stays the default. The low absolute scores are inflated downward by self-contamination
+(each task's `NATIVE_CONTEXT=4` window pulls the previous task's command/summary) — a
+real deployment effect, but it means the suite's job is *relative* regression
+detection, not an absolute capability grade.
+
+Dominant failure modes the summaries exposed, and where they point next:
 
 | failure mode | example | harness-addressable? |
 |---|---|---|
-| `<native>` tag leak in summary | `<native> Cloned repository…` | yes — strip the tag |
-| bare tool-call-as-text | `write_file ./wc.py`, `writeFile "…"` | yes — extend `_extract_text_tool_calls` beyond `<tool_call>` JSON |
+| bare tool-call-as-text | `write_file ./who.txt`, `mkdir proj`, `git clone …` (esp. 0.5b — nearly every turn) | **yes** — extend `_extract_text_tool_calls` beyond `<tool_call>` JSON |
+| `<native>` / `<tools>` tag leak in summary | `<native> Cloned repository…`, `<tools>` | yes — strip the tag |
 | path hallucination | `/home/qwen253b/conf_count.txt`, `~/` unexpanded | partly (model) |
 | content fabrication | wrote literal `dell` instead of running whoami | no (model capability) |
 
+The first row is the clear top priority — it is the single biggest score sink across
+all four models, and it is purely a harness parse gap. The benchmark now exists to
+prove whether fixing it moves the number.
+
+**Other models worth pulling for a non-qwen data point** (different family → different
+failure modes, both with strong native tool-calling): `llama3.2:3b` (~2 GB, peer to
+the 3B) and `llama3.1:8b` (~4.7 GB, peer to the 7B but general-purpose). Probe the
+`tools` field first, then `bench/run.py --model <name>`.
+
 The first two are the clear next harness improvements; the benchmark now exists to
 prove whether they move the number.