test(ai): baseline 4 local CPU models + fix prompt-fairness bug
Capture native-harness benchmark baselines for qwen2.5 0.5b/1.5b/3b and qwen2.5-coder:7b (all probed tool-capable; deepseek-r1 and NL2SH reject the tools field). All cluster at 1-2/12 with high variance; the 7B buys no pass-rate gain at ~3x latency, so qwen2.5:3b stays the default. The single biggest score sink across every model is bare tool-call-as-text leaks — a harness parse gap, the clear next improvement. Also drop "in your home directory" from the shell prompts: it made literal- minded models create a home/ subdir (/root/home/a/b/c/...) or use ~/, which the benchmark itself surfaced. Findings doc + bench README carry the model comparison table and recommendations (llama3.2:3b / llama3.1:8b for a non-qwen data point). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
+4
-4
@@ -58,19 +58,19 @@ TASKS = [
|
||||
# ───────────────────────── shell ─────────────────────────
|
||||
Task(
|
||||
"shell-easy", "shell", "easy",
|
||||
"run whoami and write its output to a file named who.txt in your home directory",
|
||||
"run whoami and write its output to a file named who.txt",
|
||||
f'test "$(cat {HOME}/who.txt)" = root',
|
||||
),
|
||||
Task(
|
||||
"shell-medium", "shell", "medium",
|
||||
"create the nested directory a/b/c in your home directory and write the "
|
||||
"text OK into a/b/c/marker.txt",
|
||||
"create the nested directory a/b/c and write the text OK into "
|
||||
"a/b/c/marker.txt",
|
||||
f'test "$(cat {HOME}/a/b/c/marker.txt)" = OK',
|
||||
),
|
||||
Task(
|
||||
"shell-hard", "shell", "hard",
|
||||
"count how many files under /etc end in .conf and write just that number "
|
||||
"to conf_count.txt in your home directory",
|
||||
"to a file conf_count.txt",
|
||||
f'test "$(cat {HOME}/conf_count.txt)" = "$(find /etc -name "*.conf" -type f 2>/dev/null | wc -l | tr -d " ")"',
|
||||
),
|
||||
|
||||
|
||||
Reference in New Issue
Block a user