Trilltechnician/super-man
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

test(ai): correct NL2SH q09 grade after executing the command

Browse Source 
Verified grades by running the actual generated commands in a sandbox. NL2SH's
q09 (sum 3rd CSV column) was marked wrong on paper but actually outputs the
correct value (60) despite an over-engineered pipeline; regraded wrong->partial.
NL2SH weighted accuracy 78%->80%, hard-tier 50%->58%; correct count unchanged.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
leetcrypt
2026-06-09 16:40:39 -07:00
parent 220fc74eb6
commit f4060a25d3
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/performance/ai-scores.csv
+1 -1
View File
@@ -57,7 +57,7 @@ westenfelder/NL2SH|q05|hard|correct
westenfelder/NL2SH|q06|easy|correct
westenfelder/NL2SH|q07|medium|partial
westenfelder/NL2SH|q08|medium|wrong
westenfelder/NL2SH|q09|hard|wrong
westenfelder/NL2SH|q09|hard|partial
westenfelder/NL2SH|q10|easy|correct
westenfelder/NL2SH|q11|medium|correct
westenfelder/NL2SH|q12|hard|correct
1 model id difficulty verdict
57 westenfelder/NL2SH q06 easy correct
58 westenfelder/NL2SH q07 medium partial
59 westenfelder/NL2SH q08 medium wrong
60 westenfelder/NL2SH q09 hard wrong partial
61 westenfelder/NL2SH q10 easy correct
62 westenfelder/NL2SH q11 medium correct
63 westenfelder/NL2SH q12 hard correct