leetcrypt
220fc74eb6
Adds a repeatable AI-mode benchmark (ai-benchmark.sh) covering 25 queries across 8 categories and 3 difficulty tiers, plus captured latency results and manual correctness grades for qwen2.5-coder:1.5b, qwen2.5:3b, and westenfelder/NL2SH. Headline finding: on the larger suite NL2SH is most accurate (72% vs coder 40%, 3b 32%) and the only model with non-zero hard-task accuracy (50% vs coder 0%), while qwen2.5-coder is ~2x faster (median 3.0s vs 6.3s). This reverses the earlier 7-query result that favored qwen-coder overall. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
77 lines
2.5 KiB
CSV
77 lines
2.5 KiB
CSV
model|id|difficulty|verdict
|
|
qwen2.5-coder:1.5b|q01|easy|wrong
|
|
qwen2.5-coder:1.5b|q02|easy|correct
|
|
qwen2.5-coder:1.5b|q03|medium|partial
|
|
qwen2.5-coder:1.5b|q04|medium|correct
|
|
qwen2.5-coder:1.5b|q05|hard|wrong
|
|
qwen2.5-coder:1.5b|q06|easy|correct
|
|
qwen2.5-coder:1.5b|q07|medium|correct
|
|
qwen2.5-coder:1.5b|q08|medium|wrong
|
|
qwen2.5-coder:1.5b|q09|hard|wrong
|
|
qwen2.5-coder:1.5b|q10|easy|correct
|
|
qwen2.5-coder:1.5b|q11|medium|correct
|
|
qwen2.5-coder:1.5b|q12|hard|wrong
|
|
qwen2.5-coder:1.5b|q13|easy|correct
|
|
qwen2.5-coder:1.5b|q14|medium|wrong
|
|
qwen2.5-coder:1.5b|q15|hard|wrong
|
|
qwen2.5-coder:1.5b|q16|easy|partial
|
|
qwen2.5-coder:1.5b|q17|medium|wrong
|
|
qwen2.5-coder:1.5b|q18|hard|wrong
|
|
qwen2.5-coder:1.5b|q19|easy|correct
|
|
qwen2.5-coder:1.5b|q20|medium|wrong
|
|
qwen2.5-coder:1.5b|q21|easy|correct
|
|
qwen2.5-coder:1.5b|q22|medium|wrong
|
|
qwen2.5-coder:1.5b|q23|easy|partial
|
|
qwen2.5-coder:1.5b|q24|medium|correct
|
|
qwen2.5-coder:1.5b|q25|hard|wrong
|
|
qwen2.5:3b|q01|easy|wrong
|
|
qwen2.5:3b|q02|easy|partial
|
|
qwen2.5:3b|q03|medium|correct
|
|
qwen2.5:3b|q04|medium|wrong
|
|
qwen2.5:3b|q05|hard|wrong
|
|
qwen2.5:3b|q06|easy|wrong
|
|
qwen2.5:3b|q07|medium|correct
|
|
qwen2.5:3b|q08|medium|correct
|
|
qwen2.5:3b|q09|hard|wrong
|
|
qwen2.5:3b|q10|easy|partial
|
|
qwen2.5:3b|q11|medium|correct
|
|
qwen2.5:3b|q12|hard|wrong
|
|
qwen2.5:3b|q13|easy|correct
|
|
qwen2.5:3b|q14|medium|correct
|
|
qwen2.5:3b|q15|hard|wrong
|
|
qwen2.5:3b|q16|easy|correct
|
|
qwen2.5:3b|q17|medium|wrong
|
|
qwen2.5:3b|q18|hard|correct
|
|
qwen2.5:3b|q19|easy|wrong
|
|
qwen2.5:3b|q20|medium|wrong
|
|
qwen2.5:3b|q21|easy|partial
|
|
qwen2.5:3b|q22|medium|wrong
|
|
qwen2.5:3b|q23|easy|partial
|
|
qwen2.5:3b|q24|medium|wrong
|
|
qwen2.5:3b|q25|hard|wrong
|
|
westenfelder/NL2SH|q01|easy|correct
|
|
westenfelder/NL2SH|q02|easy|partial
|
|
westenfelder/NL2SH|q03|medium|correct
|
|
westenfelder/NL2SH|q04|medium|correct
|
|
westenfelder/NL2SH|q05|hard|correct
|
|
westenfelder/NL2SH|q06|easy|correct
|
|
westenfelder/NL2SH|q07|medium|partial
|
|
westenfelder/NL2SH|q08|medium|wrong
|
|
westenfelder/NL2SH|q09|hard|wrong
|
|
westenfelder/NL2SH|q10|easy|correct
|
|
westenfelder/NL2SH|q11|medium|correct
|
|
westenfelder/NL2SH|q12|hard|correct
|
|
westenfelder/NL2SH|q13|easy|correct
|
|
westenfelder/NL2SH|q14|medium|correct
|
|
westenfelder/NL2SH|q15|hard|wrong
|
|
westenfelder/NL2SH|q16|easy|correct
|
|
westenfelder/NL2SH|q17|medium|correct
|
|
westenfelder/NL2SH|q18|hard|correct
|
|
westenfelder/NL2SH|q19|easy|correct
|
|
westenfelder/NL2SH|q20|medium|correct
|
|
westenfelder/NL2SH|q21|easy|correct
|
|
westenfelder/NL2SH|q22|medium|partial
|
|
westenfelder/NL2SH|q23|easy|correct
|
|
westenfelder/NL2SH|q24|medium|correct
|
|
westenfelder/NL2SH|q25|hard|wrong
|