feat(bench): multi-language capability benchmark + model picker
Add bench-lang.py + bench/ package: a third benchmark axis answering "which open-source model is best for my workflow?" across Python, JavaScript, Go, Rust and Bash. - MultiPL-E (Go/Rust/JS/Bash) + original HumanEval (Python), loaded via the HF datasets-server REST API with on-disk cache — no datasets/ pyarrow dependency. - Completions go straight to Ollama /api/generate with raw=True so instruct models continue the code instead of replying with prose. - Code runs in rootless, network-less podman (safe default) with a host-toolchain fallback; pass@1/pass@k via the HumanEval estimator. - run/score separation: results persist to a scorecard JSON, then `pick --workflow ops` re-ranks without re-running any model. - Extensible: a new language is one Lang entry; a new workflow is one block in workflows.json. Also fix a --runs grant-persistence bug in bench-sandbox.py: the grant leaked across runs, invalidating the L0-nogrant refusal test on runs 2+. Each run now revokes the ACL and starts ungranted. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,31 @@
|
||||
#!/usr/bin/env python3
|
||||
"""bench-lang.py — launcher for the multi-language capability benchmark + picker.
|
||||
|
||||
This is the third hack-house benchmark, complementing:
|
||||
• bench-ai.py — /ai chat latency/throughput on the real relay path
|
||||
• bench-sandbox.py — /ai !task sandbox code-execution + safety guards
|
||||
|
||||
bench-lang answers the capability question MultiPL-E was built for: *can this
|
||||
model actually write correct code in my language?* across Python, JavaScript,
|
||||
Go, Rust and Bash — then weights the result by your workflow to recommend a
|
||||
model. The implementation lives in the `bench/` package next to this file.
|
||||
|
||||
Examples:
|
||||
.venv/bin/python hh/scripts/bench-lang.py langs
|
||||
.venv/bin/python hh/scripts/bench-lang.py run \
|
||||
--models qwen2.5-coder:3b qwen2.5:3b --languages python bash --limit 10
|
||||
.venv/bin/python hh/scripts/bench-lang.py pick --workflow ops
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# Make the sibling `bench/` package importable when run as a plain script.
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
|
||||
from bench.cli import main # noqa: E402
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in New Issue
Block a user