leetcrypt
58d405c518
Capture native-harness benchmark baselines for qwen2.5 0.5b/1.5b/3b and qwen2.5-coder:7b (all probed tool-capable; deepseek-r1 and NL2SH reject the tools field). All cluster at 1-2/12 with high variance; the 7B buys no pass-rate gain at ~3x latency, so qwen2.5:3b stays the default. The single biggest score sink across every model is bare tool-call-as-text leaks — a harness parse gap, the clear next improvement. Also drop "in your home directory" from the shell prompts: it made literal- minded models create a home/ subdir (/root/home/a/b/c/...) or use ~/, which the benchmark itself surfaced. Findings doc + bench README carry the model comparison table and recommendations (llama3.2:3b / llama3.1:8b for a non-qwen data point). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
114 lines
5.8 KiB
JSON
114 lines
5.8 KiB
JSON
{
|
|
"ts": "2026-06-10T09:28:37",
|
|
"model": "qwen2.5:3b",
|
|
"rows": [
|
|
{
|
|
"id": "shell-easy",
|
|
"category": "shell",
|
|
"tier": "easy",
|
|
"result": "FAIL",
|
|
"secs": 89.0,
|
|
"summary": "run_shell './who.txt | tee who.out' \u2502\u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
|
|
"verify_out": ""
|
|
},
|
|
{
|
|
"id": "shell-medium",
|
|
"category": "shell",
|
|
"tier": "medium",
|
|
"result": "FAIL",
|
|
"secs": 32.7,
|
|
"summary": "(write_file \"a/b/c/marker.txt\" \"OK\") \u2502\u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
|
|
"verify_out": "cat: /root/a/b/c/marker.txt: No such file or directory"
|
|
},
|
|
{
|
|
"id": "shell-hard",
|
|
"category": "shell",
|
|
"tier": "hard",
|
|
"result": "FAIL",
|
|
"secs": 32.7,
|
|
"summary": "5 \u2502\u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
|
|
"verify_out": "cat: /root/conf_count.txt: No such file or directory"
|
|
},
|
|
{
|
|
"id": "code-easy",
|
|
"category": "code",
|
|
"tier": "easy",
|
|
"result": "PASS",
|
|
"secs": 40.7,
|
|
"summary": "Successfully wrote `add.py`, ran it with Python 3, and saved the output to `add.out`.\u2502\u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
|
|
"verify_out": ""
|
|
},
|
|
{
|
|
"id": "code-medium",
|
|
"category": "code",
|
|
"tier": "medium",
|
|
"result": "FAIL",
|
|
"secs": 36.7,
|
|
"summary": "Successfully wrote `add.py`, ran it with Python 3, and saved the output to `add.out`.\u2502\u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
|
|
"verify_out": ""
|
|
},
|
|
{
|
|
"id": "code-hard",
|
|
"category": "code",
|
|
"tier": "hard",
|
|
"result": "FAIL",
|
|
"secs": 42.7,
|
|
"summary": "The task is incomplete. Please provide additional instructions or complete the task \u2502\u2502 \u2502 \u2502before calling another tool.",
|
|
"verify_out": "cat: /root/fib.out: No such file or directory"
|
|
},
|
|
{
|
|
"id": "git-easy",
|
|
"category": "git",
|
|
"tier": "easy",
|
|
"result": "FAIL",
|
|
"secs": 36.7,
|
|
"summary": "",
|
|
"verify_out": ""
|
|
},
|
|
{
|
|
"id": "git-medium",
|
|
"category": "git",
|
|
"tier": "medium",
|
|
"result": "FAIL",
|
|
"secs": 56.8,
|
|
"summary": "{\"name\": \"write_file\", \"arguments\": \"{\\\"path\\\": \\\"hw2/README.md\\\", \\\"content\\\": \\\"# \u2502\u2502 \u2502 \u2502Hello World\\nThis is a simple example of how to st",
|
|
"verify_out": "grep: /root/readme.txt: No such file or directory"
|
|
},
|
|
{
|
|
"id": "git-hard",
|
|
"category": "git",
|
|
"tier": "hard",
|
|
"result": "FAIL",
|
|
"secs": 48.8,
|
|
"summary": "{\"name\": \"run_shell\", \"arguments\": \"{\\\"command\\\": \\\"git clone \u2502\u2502 \u2502 \u2502https://github.com/octocat/Hello-World hw3 && cd h",
|
|
"verify_out": "sh: 1: cannot open /root/branch.txt: No such file"
|
|
},
|
|
{
|
|
"id": "multi-easy",
|
|
"category": "multi",
|
|
"tier": "easy",
|
|
"result": "FAIL",
|
|
"secs": 66.8,
|
|
"summary": "./proj contains 1 file(s) with content:",
|
|
"verify_out": "cat: /root/proj/notes.txt: No such file or directory"
|
|
},
|
|
{
|
|
"id": "multi-medium",
|
|
"category": "multi",
|
|
"tier": "medium",
|
|
"result": "FAIL",
|
|
"secs": 58.8,
|
|
"summary": "{\"name\": \"write_file\", \"arguments\": \"{\\\"path\\\": \\\"fruit.txt\\\", \\\"content\\\": \u2502\u2502 \u2502 \u2502\\\"apple\\nbanana\\ cherry\\\"}\"}",
|
|
"verify_out": "cat: /root/sorted.txt: No such file or directory"
|
|
},
|
|
{
|
|
"id": "multi-hard",
|
|
"category": "multi",
|
|
"tier": "hard",
|
|
"result": "FAIL",
|
|
"secs": 64.8,
|
|
"summary": "3 \u2502\u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
|
|
"verify_out": "sh: 1: cannot open /root/wc.out: No such file"
|
|
}
|
|
]
|
|
} |