Add bench-lang.py + bench/ package: a third benchmark axis answering
"which open-source model is best for my workflow?" across Python,
JavaScript, Go, Rust and Bash.
- MultiPL-E (Go/Rust/JS/Bash) + original HumanEval (Python), loaded via
the HF datasets-server REST API with on-disk cache — no datasets/
pyarrow dependency.
- Completions go straight to Ollama /api/generate with raw=True so
instruct models continue the code instead of replying with prose.
- Code runs in rootless, network-less podman (safe default) with a
host-toolchain fallback; pass@1/pass@k via the HumanEval estimator.
- run/score separation: results persist to a scorecard JSON, then
`pick --workflow ops` re-ranks without re-running any model.
- Extensible: a new language is one Lang entry; a new workflow is one
block in workflows.json.
Also fix a --runs grant-persistence bug in bench-sandbox.py: the grant
leaked across runs, invalidating the L0-nogrant refusal test on runs 2+.
Each run now revokes the ACL and starts ungranted.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
leetcrypt
7fb3911550