test(ai): baseline 4 local CPU models + fix prompt-fairness bug

Capture native-harness benchmark baselines for qwen2.5 0.5b/1.5b/3b and
qwen2.5-coder:7b (all probed tool-capable; deepseek-r1 and NL2SH reject the
tools field). All cluster at 1-2/12 with high variance; the 7B buys no
pass-rate gain at ~3x latency, so qwen2.5:3b stays the default. The single
biggest score sink across every model is bare tool-call-as-text leaks — a
harness parse gap, the clear next improvement.

Also drop "in your home directory" from the shell prompts: it made literal-
minded models create a home/ subdir (/root/home/a/b/c/...) or use ~/, which
the benchmark itself surfaced. Findings doc + bench README carry the model
comparison table and recommendations (llama3.2:3b / llama3.1:8b for a non-qwen
data point).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
leetcrypt
2026-06-10 09:30:36 -07:00
parent df45303f5c
commit 58d405c518
7 changed files with 434 additions and 44 deletions
+4 -4
View File
@@ -58,19 +58,19 @@ TASKS = [
# ───────────────────────── shell ─────────────────────────
Task(
"shell-easy", "shell", "easy",
"run whoami and write its output to a file named who.txt in your home directory",
"run whoami and write its output to a file named who.txt",
f'test "$(cat {HOME}/who.txt)" = root',
),
Task(
"shell-medium", "shell", "medium",
"create the nested directory a/b/c in your home directory and write the "
"text OK into a/b/c/marker.txt",
"create the nested directory a/b/c and write the text OK into "
"a/b/c/marker.txt",
f'test "$(cat {HOME}/a/b/c/marker.txt)" = OK',
),
Task(
"shell-hard", "shell", "hard",
"count how many files under /etc end in .conf and write just that number "
"to conf_count.txt in your home directory",
"to a file conf_count.txt",
f'test "$(cat {HOME}/conf_count.txt)" = "$(find /etc -name "*.conf" -type f 2>/dev/null | wc -l | tr -d " ")"',
),