Trilltechnician/hack-house
1
3
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

test(ai): baseline 4 local CPU models + fix prompt-fairness bug

Browse Source 
Capture native-harness benchmark baselines for qwen2.5 0.5b/1.5b/3b and
qwen2.5-coder:7b (all probed tool-capable; deepseek-r1 and NL2SH reject the
tools field). All cluster at 1-2/12 with high variance; the 7B buys no
pass-rate gain at ~3x latency, so qwen2.5:3b stays the default. The single
biggest score sink across every model is bare tool-call-as-text leaks — a
harness parse gap, the clear next improvement.

Also drop "in your home directory" from the shell prompts: it made literal-
minded models create a home/ subdir (/root/home/a/b/c/...) or use ~/, which
the benchmark itself surfaced. Findings doc + bench README carry the model
comparison table and recommendations (llama3.2:3b / llama3.1:8b for a non-qwen
data point).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
leetcrypt
2026-06-10 09:30:36 -07:00
parent df45303f5c
commit 58d405c518
7 changed files with 434 additions and 44 deletions
Download Patch File Download Diff File Expand all files Collapse all files
bench/results/baseline-qwen2.5-coder_1.5b.json
+114
View File
@@ -0,0 +1,114 @@
{
"ts": "2026-06-10T09:03:58",
"model": "qwen2.5-coder:1.5b",
"rows": [
{
"id": "shell-easy",
"category": "shell",
"tier": "easy",
"result": "FAIL",
"secs": 144.0,
"summary": "wc.py ran and saved output to wc.out, whoami output written to who.txt. \u2502\u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
"verify_out": ""
},
{
"id": "shell-medium",
"category": "shell",
"tier": "medium",
"result": "PASS",
"secs": 48.8,
"summary": ".",
"verify_out": ""
},
{
"id": "shell-hard",
"category": "shell",
"tier": "hard",
"result": "FAIL",
"secs": 40.8,
"summary": ". \u2502\u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
"verify_out": "cat: /root/conf_count.txt: No such file or directory"
},
{
"id": "code-easy",
"category": "code",
"tier": "easy",
"result": "FAIL",
"secs": 52.9,
"summary": ".",
"verify_out": "cat: /root/add.out: No such file or directory"
},
{
"id": "code-medium",
"category": "code",
"tier": "medium",
"result": "FAIL",
"secs": 71.1,
"summary": "chmod command failed; check permissions for the script file. \u2502\u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
"verify_out": ""
},
{
"id": "code-hard",
"category": "code",
"tier": "hard",
"result": "FAIL",
"secs": 46.8,
"summary": "write_file ./fib.py 'print(\"0 1\") for i in range(10); print(\" \".join(map(str,\u2502\u2502 \u2502 \u2502[i+j for i,j in zip([0,1], [1,2])))))'",
"verify_out": "cat: /root/fib.out: No such file or directory"
},
{
"id": "git-easy",
"category": "git",
"tier": "easy",
"result": "FAIL",
"secs": 40.9,
"summary": "",
"verify_out": ""
},
{
"id": "git-medium",
"category": "git",
"tier": "medium",
"result": "FAIL",
"secs": 46.8,
"summary": "run_shell 'git clone https://github.com/octocat/Hello-World' \u2502\u2502 \u2502 \u2502 DONE: run_shell 'mkdir hw2 && cd h",
"verify_out": "grep: /root/readme.txt: No such file or directory"
},
{
"id": "git-hard",
"category": "git",
"tier": "hard",
"result": "FAIL",
"secs": 54.9,
"summary": "run_shell 'git clone https://github.com/octocat/Hello-World' \u2502\u2502 \u2502 \u2502 DONE: run_shell 'mkdir hw3 && cd h",
"verify_out": "sh: 1: cannot open /root/branch.txt: No such file"
},
{
"id": "multi-easy",
"category": "multi",
"tier": "easy",
"result": "FAIL",
"secs": 52.9,
"summary": "mkdir proj \u2502\u2502 \u2502 \u2502 DONE: write_file ./proj/notes.txt ",
"verify_out": "cat: /root/proj/notes.txt: No such file or directory"
},
{
"id": "multi-medium",
"category": "multi",
"tier": "medium",
"result": "FAIL",
"secs": 99.2,
"summary": "",
"verify_out": "cat: /root/sorted.txt: No such file or directory"
},
{
"id": "multi-hard",
"category": "multi",
"tier": "hard",
"result": "FAIL",
"secs": 107.5,
"summary": "write_file 'one\\ntwo\\tthree' ./words.txt",
"verify_out": ""
}
]
}
bench/results/baseline-qwen2.5-coder_7b.json
+114
View File
@@ -0,0 +1,114 @@
{
"ts": "2026-06-10T08:48:26",
"model": "qwen2.5-coder:7b",
"rows": [
{
"id": "shell-easy",
"category": "shell",
"tier": "easy",
"result": "PASS",
"secs": 70.9,
"summary": "",
"verify_out": ""
},
{
"id": "shell-medium",
"category": "shell",
"tier": "medium",
"result": "FAIL",
"secs": 67.0,
"summary": "",
"verify_out": "cat: /root/a/b/c/marker.txt: No such file or directory"
},
{
"id": "shell-hard",
"category": "shell",
"tier": "hard",
"result": "FAIL",
"secs": 60.9,
"summary": "(done) \u2502\u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
"verify_out": "cat: /root/conf_count.txt: No such file or directory"
},
{
"id": "code-easy",
"category": "code",
"tier": "easy",
"result": "FAIL",
"secs": 210.0,
"summary": "",
"verify_out": "cat: /root/add.out: No such file or directory"
},
{
"id": "code-medium",
"category": "code",
"tier": "medium",
"result": "FAIL",
"secs": 274.3,
"summary": "",
"verify_out": ""
},
{
"id": "code-hard",
"category": "code",
"tier": "hard",
"result": "FAIL",
"secs": 109.0,
"summary": "<tools> \u2502\u2502 \u2502 \u2502 {\"type\": \"function\", \"function\": {",
"verify_out": "cat: /root/fib.out: No such file or directory"
},
{
"id": "git-easy",
"category": "git",
"tier": "easy",
"result": "FAIL",
"secs": 103.3,
"summary": "[stopped \u2014 model described the task but never ran a tool] \u2502\u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
"verify_out": ""
},
{
"id": "git-medium",
"category": "git",
"tier": "medium",
"result": "FAIL",
"secs": 204.0,
"summary": "",
"verify_out": "grep: /root/readme.txt: No such file or directory"
},
{
"id": "git-hard",
"category": "git",
"tier": "hard",
"result": "FAIL",
"secs": 103.4,
"summary": "<tools> \u2502\u2502 \u2502 \u2502 {\"type\": \"function\", \"function\": {",
"verify_out": ""
},
{
"id": "multi-easy",
"category": "multi",
"tier": "easy",
"result": "PASS",
"secs": 101.8,
"summary": "",
"verify_out": ""
},
{
"id": "multi-medium",
"category": "multi",
"tier": "medium",
"result": "FAIL",
"secs": 191.3,
"summary": "(native) . \u2502\u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
"verify_out": ""
},
{
"id": "multi-hard",
"category": "multi",
"tier": "hard",
"result": "FAIL",
"secs": 192.8,
"summary": "",
"verify_out": ""
}
]
}
bench/results/baseline-qwen2.5_0.5b.json
+114
View File
@@ -0,0 +1,114 @@
{
"ts": "2026-06-10T09:16:10",
"model": "qwen2.5:0.5b",
"rows": [
{
"id": "shell-easy",
"category": "shell",
"tier": "easy",
"result": "FAIL",
"secs": 151.9,
"summary": "write_file ./who.txt \u2502\u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
"verify_out": ""
},
{
"id": "shell-medium",
"category": "shell",
"tier": "medium",
"result": "FAIL",
"secs": 52.9,
"summary": "write_file ./a/b/c/marker.txt \"OK\" \u2502\u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
"verify_out": "cat: /root/a/b/c/marker.txt: No such file or directory"
},
{
"id": "shell-hard",
"category": "shell",
"tier": "hard",
"result": "PASS",
"secs": 48.8,
"summary": "Successfully wrote the count to conf_count.txt \u2502\u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
"verify_out": ""
},
{
"id": "code-easy",
"category": "code",
"tier": "easy",
"result": "FAIL",
"secs": 36.7,
"summary": "",
"verify_out": "cat: /root/add.out: No such file or directory"
},
{
"id": "code-medium",
"category": "code",
"tier": "medium",
"result": "FAIL",
"secs": 40.7,
"summary": "write_file ./greet.sh",
"verify_out": ""
},
{
"id": "code-hard",
"category": "code",
"tier": "hard",
"result": "FAIL",
"secs": 48.8,
"summary": "fib.py written and chmod +x'd",
"verify_out": "cat: /root/fib.out: No such file or directory"
},
{
"id": "git-easy",
"category": "git",
"tier": "easy",
"result": "FAIL",
"secs": 32.7,
"summary": "",
"verify_out": ""
},
{
"id": "git-medium",
"category": "git",
"tier": "medium",
"result": "FAIL",
"secs": 40.7,
"summary": "read_file ./hw2/README.md > readme.txt \u2502\u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
"verify_out": "grep: /root/readme.txt: No such file or directory"
},
{
"id": "git-hard",
"category": "git",
"tier": "hard",
"result": "FAIL",
"secs": 34.7,
"summary": "git clone https://github.com/octocat/Hello-World hw3",
"verify_out": "sh: 1: cannot open /root/branch.txt: No such file"
},
{
"id": "multi-easy",
"category": "multi",
"tier": "easy",
"result": "FAIL",
"secs": 48.9,
"summary": "mkdir proj",
"verify_out": "cat: /root/proj/notes.txt: No such file or directory"
},
{
"id": "multi-medium",
"category": "multi",
"tier": "medium",
"result": "FAIL",
"secs": 75.0,
"summary": "write_file ./fruit.txt \"apple banana cherry\"",
"verify_out": "cat: /root/sorted.txt: No such file or directory"
},
{
"id": "multi-hard",
"category": "multi",
"tier": "hard",
"result": "FAIL",
"secs": 52.8,
"summary": "wc.py < words.txt > wc.out \u2502\u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
"verify_out": "sh: 1: cannot open /root/wc.out: No such file"
}
]
}
bench/results/baseline-qwen2.5_3b.json
+32 -32
View File
@@ -1,5 +1,5 @@
{
"ts": "2026-06-10T01:29:10",
"ts": "2026-06-10T09:28:37",
"model": "qwen2.5:3b",
"rows": [
{
@@ -7,44 +7,44 @@
"category": "shell",
"tier": "easy",
"result": "FAIL",
"secs": 38.7,
"summary": "wrote_file \"~/who.txt\" \"dell\", run_shell \"ls -a ~/who.txt\" \u2502\u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
"verify_out": "cat: /root/who.txt: No such file or directory"
"secs": 89.0,
"summary": "run_shell './who.txt | tee who.out' \u2502\u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
"verify_out": ""
},
{
"id": "shell-medium",
"category": "shell",
"tier": "medium",
"result": "PASS",
"secs": 34.7,
"summary": "Created nested directory 'a/b/c' and wrote \"OK\" into 'marker.txt'. \u2502\u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
"verify_out": ""
"result": "FAIL",
"secs": 32.7,
"summary": "(write_file \"a/b/c/marker.txt\" \"OK\") \u2502\u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
"verify_out": "cat: /root/a/b/c/marker.txt: No such file or directory"
},
{
"id": "shell-hard",
"category": "shell",
"tier": "hard",
"result": "FAIL",
"secs": 28.7,
"summary": "writeFile \"/home/qwen253b/conf_count.txt\" \"$(grep -c \"\\.conf$\" /etc)\" \u2502\u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
"secs": 32.7,
"summary": "5 \u2502\u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
"verify_out": "cat: /root/conf_count.txt: No such file or directory"
},
{
"id": "code-easy",
"category": "code",
"tier": "easy",
"result": "FAIL",
"secs": 34.7,
"summary": "Added Python script add.py and ran it. \u2502\u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
"verify_out": "cat: /root/add.out: No such file or directory"
"result": "PASS",
"secs": 40.7,
"summary": "Successfully wrote `add.py`, ran it with Python 3, and saved the output to `add.out`.\u2502\u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
"verify_out": ""
},
{
"id": "code-medium",
"category": "code",
"tier": "medium",
"result": "FAIL",
"secs": 48.9,
"summary": "Script greet.sh has been created, made executable, and ran successfully. The output \u2502\u2502 \u2502 \u2502is saved in greet.out.",
"secs": 36.7,
"summary": "Successfully wrote `add.py`, ran it with Python 3, and saved the output to `add.out`.\u2502\u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
"verify_out": ""
},
{
@@ -52,8 +52,8 @@
"category": "code",
"tier": "hard",
"result": "FAIL",
"secs": 44.8,
"summary": "<native> Created file 'fib.py' with contents \"def fib(n): if n <= 1: return\u2502\u2502 \u2502 \u2502n else: a, b = 0, 1 for i in rang",
"secs": 42.7,
"summary": "The task is incomplete. Please provide additional instructions or complete the task \u2502\u2502 \u2502 \u2502before calling another tool.",
"verify_out": "cat: /root/fib.out: No such file or directory"
},
{
@@ -61,8 +61,8 @@
"category": "git",
"tier": "easy",
"result": "FAIL",
"secs": 32.7,
"summary": "<native> Cloned repository 'https://github.com/octocat/Hello-World' into './hw'. \u2502\u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
"secs": 36.7,
"summary": "",
"verify_out": ""
},
{
@@ -70,8 +70,8 @@
"category": "git",
"tier": "medium",
"result": "FAIL",
"secs": 44.8,
"summary": "(native) <native> Cloned repository 'https://github.com/octocat/Hello-World' into \u2502\u2502 \u271d\u26a1\u25c6 dell \u2502 \u2502'./hw2'. ",
"secs": 56.8,
"summary": "{\"name\": \"write_file\", \"arguments\": \"{\\\"path\\\": \\\"hw2/README.md\\\", \\\"content\\\": \\\"# \u2502\u2502 \u2502 \u2502Hello World\\nThis is a simple example of how to st",
"verify_out": "grep: /root/readme.txt: No such file or directory"
},
{
@@ -79,26 +79,26 @@
"category": "git",
"tier": "hard",
"result": "FAIL",
"secs": 32.7,
"summary": "<native> Cloned repository 'https://github.com/octocat/Hello-World' into './hw3'.",
"secs": 48.8,
"summary": "{\"name\": \"run_shell\", \"arguments\": \"{\\\"command\\\": \\\"git clone \u2502\u2502 \u2502 \u2502https://github.com/octocat/Hello-World hw3 && cd h",
"verify_out": "sh: 1: cannot open /root/branch.txt: No such file"
},
{
"id": "multi-easy",
"category": "multi",
"tier": "easy",
"result": "PASS",
"secs": 44.8,
"summary": "<native> Made 'proj' directory and created 'notes.txt' with content \"hello world\". \u2502\u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
"verify_out": ""
"result": "FAIL",
"secs": 66.8,
"summary": "./proj contains 1 file(s) with content:",
"verify_out": "cat: /root/proj/notes.txt: No such file or directory"
},
{
"id": "multi-medium",
"category": "multi",
"tier": "medium",
"result": "FAIL",
"secs": 42.8,
"summary": "<native> Created file 'sorted.txt' from content in 'fruit.txt'.",
"secs": 58.8,
"summary": "{\"name\": \"write_file\", \"arguments\": \"{\\\"path\\\": \\\"fruit.txt\\\", \\\"content\\\": \u2502\u2502 \u2502 \u2502\\\"apple\\nbanana\\ cherry\\\"}\"}",
"verify_out": "cat: /root/sorted.txt: No such file or directory"
},
{
@@ -106,8 +106,8 @@
"category": "multi",
"tier": "hard",
"result": "FAIL",
"secs": 54.8,
"summary": "<native> Executed `write_file ./wc.py \u2502\u2502 \u2502 \u2502\"print(len(open('./words.txt').read().split()))\"` ",
"secs": 64.8,
"summary": "3 \u2502\u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500",
"verify_out": "sh: 1: cannot open /root/wc.out: No such file"
}
]