Verified grades by running the actual generated commands in a sandbox. NL2SH's
q09 (sum 3rd CSV column) was marked wrong on paper but actually outputs the
correct value (60) despite an over-engineered pipeline; regraded wrong->partial.
NL2SH weighted accuracy 78%->80%, hard-tier 50%->58%; correct count unchanged.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds a repeatable AI-mode benchmark (ai-benchmark.sh) covering 25 queries across
8 categories and 3 difficulty tiers, plus captured latency results and manual
correctness grades for qwen2.5-coder:1.5b, qwen2.5:3b, and westenfelder/NL2SH.
Headline finding: on the larger suite NL2SH is most accurate (72% vs coder 40%,
3b 32%) and the only model with non-zero hard-task accuracy (50% vs coder 0%),
while qwen2.5-coder is ~2x faster (median 3.0s vs 6.3s). This reverses the
earlier 7-query result that favored qwen-coder overall.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
leetcrypt