Trilltechnician/super-man
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

test(ai): add 25-query benchmark harness + results for ai mode models

Browse Source 
Adds a repeatable AI-mode benchmark (ai-benchmark.sh) covering 25 queries across
8 categories and 3 difficulty tiers, plus captured latency results and manual
correctness grades for qwen2.5-coder:1.5b, qwen2.5:3b, and westenfelder/NL2SH.

Headline finding: on the larger suite NL2SH is most accurate (72% vs coder 40%,
3b 32%) and the only model with non-zero hard-task accuracy (50% vs coder 0%),
while qwen2.5-coder is ~2x faster (median 3.0s vs 6.3s). This reverses the
earlier 7-query result that favored qwen-coder overall.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
leetcrypt
2026-06-09 15:53:24 -07:00
parent e4149ce502
commit 220fc74eb6
3 changed files with 225 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/performance/ai-scores.csv
+76
View File
@@ -0,0 +1,76 @@
model|id|difficulty|verdict
qwen2.5-coder:1.5b|q01|easy|wrong
qwen2.5-coder:1.5b|q02|easy|correct
qwen2.5-coder:1.5b|q03|medium|partial
qwen2.5-coder:1.5b|q04|medium|correct
qwen2.5-coder:1.5b|q05|hard|wrong
qwen2.5-coder:1.5b|q06|easy|correct
qwen2.5-coder:1.5b|q07|medium|correct
qwen2.5-coder:1.5b|q08|medium|wrong
qwen2.5-coder:1.5b|q09|hard|wrong
qwen2.5-coder:1.5b|q10|easy|correct
qwen2.5-coder:1.5b|q11|medium|correct
qwen2.5-coder:1.5b|q12|hard|wrong
qwen2.5-coder:1.5b|q13|easy|correct
qwen2.5-coder:1.5b|q14|medium|wrong
qwen2.5-coder:1.5b|q15|hard|wrong
qwen2.5-coder:1.5b|q16|easy|partial
qwen2.5-coder:1.5b|q17|medium|wrong
qwen2.5-coder:1.5b|q18|hard|wrong
qwen2.5-coder:1.5b|q19|easy|correct
qwen2.5-coder:1.5b|q20|medium|wrong
qwen2.5-coder:1.5b|q21|easy|correct
qwen2.5-coder:1.5b|q22|medium|wrong
qwen2.5-coder:1.5b|q23|easy|partial
qwen2.5-coder:1.5b|q24|medium|correct
qwen2.5-coder:1.5b|q25|hard|wrong
qwen2.5:3b|q01|easy|wrong
qwen2.5:3b|q02|easy|partial
qwen2.5:3b|q03|medium|correct
qwen2.5:3b|q04|medium|wrong
qwen2.5:3b|q05|hard|wrong
qwen2.5:3b|q06|easy|wrong
qwen2.5:3b|q07|medium|correct
qwen2.5:3b|q08|medium|correct
qwen2.5:3b|q09|hard|wrong
qwen2.5:3b|q10|easy|partial
qwen2.5:3b|q11|medium|correct
qwen2.5:3b|q12|hard|wrong
qwen2.5:3b|q13|easy|correct
qwen2.5:3b|q14|medium|correct
qwen2.5:3b|q15|hard|wrong
qwen2.5:3b|q16|easy|correct
qwen2.5:3b|q17|medium|wrong
qwen2.5:3b|q18|hard|correct
qwen2.5:3b|q19|easy|wrong
qwen2.5:3b|q20|medium|wrong
qwen2.5:3b|q21|easy|partial
qwen2.5:3b|q22|medium|wrong
qwen2.5:3b|q23|easy|partial
qwen2.5:3b|q24|medium|wrong
qwen2.5:3b|q25|hard|wrong
westenfelder/NL2SH|q01|easy|correct
westenfelder/NL2SH|q02|easy|partial
westenfelder/NL2SH|q03|medium|correct
westenfelder/NL2SH|q04|medium|correct
westenfelder/NL2SH|q05|hard|correct
westenfelder/NL2SH|q06|easy|correct
westenfelder/NL2SH|q07|medium|partial
westenfelder/NL2SH|q08|medium|wrong
westenfelder/NL2SH|q09|hard|wrong
westenfelder/NL2SH|q10|easy|correct
westenfelder/NL2SH|q11|medium|correct
westenfelder/NL2SH|q12|hard|correct
westenfelder/NL2SH|q13|easy|correct
westenfelder/NL2SH|q14|medium|correct
westenfelder/NL2SH|q15|hard|wrong
westenfelder/NL2SH|q16|easy|correct
westenfelder/NL2SH|q17|medium|correct
westenfelder/NL2SH|q18|hard|correct
westenfelder/NL2SH|q19|easy|correct
westenfelder/NL2SH|q20|medium|correct
westenfelder/NL2SH|q21|easy|correct
westenfelder/NL2SH|q22|medium|partial
westenfelder/NL2SH|q23|easy|correct
westenfelder/NL2SH|q24|medium|correct
westenfelder/NL2SH|q25|hard|wrong
1 model id difficulty verdict
2 qwen2.5-coder:1.5b q01 easy wrong
3 qwen2.5-coder:1.5b q02 easy correct
4 qwen2.5-coder:1.5b q03 medium partial
5 qwen2.5-coder:1.5b q04 medium correct
6 qwen2.5-coder:1.5b q05 hard wrong
7 qwen2.5-coder:1.5b q06 easy correct
8 qwen2.5-coder:1.5b q07 medium correct
9 qwen2.5-coder:1.5b q08 medium wrong
10 qwen2.5-coder:1.5b q09 hard wrong
11 qwen2.5-coder:1.5b q10 easy correct
12 qwen2.5-coder:1.5b q11 medium correct
13 qwen2.5-coder:1.5b q12 hard wrong
14 qwen2.5-coder:1.5b q13 easy correct
15 qwen2.5-coder:1.5b q14 medium wrong
16 qwen2.5-coder:1.5b q15 hard wrong
17 qwen2.5-coder:1.5b q16 easy partial
18 qwen2.5-coder:1.5b q17 medium wrong
19 qwen2.5-coder:1.5b q18 hard wrong
20 qwen2.5-coder:1.5b q19 easy correct
21 qwen2.5-coder:1.5b q20 medium wrong
22 qwen2.5-coder:1.5b q21 easy correct
23 qwen2.5-coder:1.5b q22 medium wrong
24 qwen2.5-coder:1.5b q23 easy partial
25 qwen2.5-coder:1.5b q24 medium correct
26 qwen2.5-coder:1.5b q25 hard wrong
27 qwen2.5:3b q01 easy wrong
28 qwen2.5:3b q02 easy partial
29 qwen2.5:3b q03 medium correct
30 qwen2.5:3b q04 medium wrong
31 qwen2.5:3b q05 hard wrong
32 qwen2.5:3b q06 easy wrong
33 qwen2.5:3b q07 medium correct
34 qwen2.5:3b q08 medium correct
35 qwen2.5:3b q09 hard wrong
36 qwen2.5:3b q10 easy partial
37 qwen2.5:3b q11 medium correct
38 qwen2.5:3b q12 hard wrong
39 qwen2.5:3b q13 easy correct
40 qwen2.5:3b q14 medium correct
41 qwen2.5:3b q15 hard wrong
42 qwen2.5:3b q16 easy correct
43 qwen2.5:3b q17 medium wrong
44 qwen2.5:3b q18 hard correct
45 qwen2.5:3b q19 easy wrong
46 qwen2.5:3b q20 medium wrong
47 qwen2.5:3b q21 easy partial
48 qwen2.5:3b q22 medium wrong
49 qwen2.5:3b q23 easy partial
50 qwen2.5:3b q24 medium wrong
51 qwen2.5:3b q25 hard wrong
52 westenfelder/NL2SH q01 easy correct
53 westenfelder/NL2SH q02 easy partial
54 westenfelder/NL2SH q03 medium correct
55 westenfelder/NL2SH q04 medium correct
56 westenfelder/NL2SH q05 hard correct
57 westenfelder/NL2SH q06 easy correct
58 westenfelder/NL2SH q07 medium partial
59 westenfelder/NL2SH q08 medium wrong
60 westenfelder/NL2SH q09 hard wrong
61 westenfelder/NL2SH q10 easy correct
62 westenfelder/NL2SH q11 medium correct
63 westenfelder/NL2SH q12 hard correct
64 westenfelder/NL2SH q13 easy correct
65 westenfelder/NL2SH q14 medium correct
66 westenfelder/NL2SH q15 hard wrong
67 westenfelder/NL2SH q16 easy correct
68 westenfelder/NL2SH q17 medium correct
69 westenfelder/NL2SH q18 hard correct
70 westenfelder/NL2SH q19 easy correct
71 westenfelder/NL2SH q20 medium correct
72 westenfelder/NL2SH q21 easy correct
73 westenfelder/NL2SH q22 medium partial
74 westenfelder/NL2SH q23 easy correct
75 westenfelder/NL2SH q24 medium correct
76 westenfelder/NL2SH q25 hard wrong