leetcrypt
7fb3911550
Add bench-lang.py + bench/ package: a third benchmark axis answering "which open-source model is best for my workflow?" across Python, JavaScript, Go, Rust and Bash. - MultiPL-E (Go/Rust/JS/Bash) + original HumanEval (Python), loaded via the HF datasets-server REST API with on-disk cache — no datasets/ pyarrow dependency. - Completions go straight to Ollama /api/generate with raw=True so instruct models continue the code instead of replying with prose. - Code runs in rootless, network-less podman (safe default) with a host-toolchain fallback; pass@1/pass@k via the HumanEval estimator. - run/score separation: results persist to a scorecard JSON, then `pick --workflow ops` re-ranks without re-running any model. - Extensible: a new language is one Lang entry; a new workflow is one block in workflows.json. Also fix a --runs grant-persistence bug in bench-sandbox.py: the grant leaked across runs, invalidating the L0-nogrant refusal test on runs 2+. Each run now revokes the ACL and starts ungranted. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
24 lines
877 B
JSON
24 lines
877 B
JSON
{
|
|
"_comment": "Workflow profiles weight per-language capability into one score. Weights need not sum to 1; they are normalised at scoring time. Add a profile here to teach the model-picker a new kind of user.",
|
|
"balanced": {
|
|
"label": "Balanced polyglot",
|
|
"weights": {"python": 1, "javascript": 1, "go": 1, "rust": 1, "bash": 1}
|
|
},
|
|
"ops": {
|
|
"label": "Ops / shell automation",
|
|
"weights": {"bash": 3, "python": 2, "go": 1, "javascript": 0.5, "rust": 0.5}
|
|
},
|
|
"backend": {
|
|
"label": "Backend services",
|
|
"weights": {"go": 3, "rust": 2, "python": 2, "javascript": 1, "bash": 1}
|
|
},
|
|
"webdev": {
|
|
"label": "Web development",
|
|
"weights": {"javascript": 3, "python": 2, "bash": 1, "go": 1, "rust": 0.5}
|
|
},
|
|
"systems": {
|
|
"label": "Systems programming",
|
|
"weights": {"rust": 3, "go": 2, "python": 1, "bash": 1, "javascript": 0.5}
|
|
}
|
|
}
|