Fourth benchmark axis: teams of LLM agents deliberate in a room, implement
code in an isolated VM, and are scored deterministically on correctness/speed.
- Multi-language adapter (python/js/go/rust/bash) via MultiPL-E continuation mode
- Append-only JSONL ledger with status tracking (ok/dnf/killed/error) so
budget-exhausted or crashed runs still record a row (fixes selection bias)
- Model-aware wall-clock scaling (U-shaped by param count; 3x for reasoning)
- Self-owned SIGALRM/SIGTERM watchdog (RunTimeout: BaseException so broad
except Exception handlers in the infer/completion path can't swallow it)
- Seed forwarded to Ollama sampler + markdown-fence stripping in completion.py
Add bench-lang.py + bench/ package: a third benchmark axis answering
"which open-source model is best for my workflow?" across Python,
JavaScript, Go, Rust and Bash.
- MultiPL-E (Go/Rust/JS/Bash) + original HumanEval (Python), loaded via
the HF datasets-server REST API with on-disk cache — no datasets/
pyarrow dependency.
- Completions go straight to Ollama /api/generate with raw=True so
instruct models continue the code instead of replying with prose.
- Code runs in rootless, network-less podman (safe default) with a
host-toolchain fallback; pass@1/pass@k via the HumanEval estimator.
- run/score separation: results persist to a scorecard JSON, then
`pick --workflow ops` re-ranks without re-running any model.
- Extensible: a new language is one Lang entry; a new workflow is one
block in workflows.json.
Also fix a --runs grant-persistence bug in bench-sandbox.py: the grant
leaked across runs, invalidating the L0-nogrant refusal test on runs 2+.
Each run now revokes the ACL and starts ungranted.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
leetcrypt