leetcrypt
7fb3911550
Add bench-lang.py + bench/ package: a third benchmark axis answering "which open-source model is best for my workflow?" across Python, JavaScript, Go, Rust and Bash. - MultiPL-E (Go/Rust/JS/Bash) + original HumanEval (Python), loaded via the HF datasets-server REST API with on-disk cache — no datasets/ pyarrow dependency. - Completions go straight to Ollama /api/generate with raw=True so instruct models continue the code instead of replying with prose. - Code runs in rootless, network-less podman (safe default) with a host-toolchain fallback; pass@1/pass@k via the HumanEval estimator. - run/score separation: results persist to a scorecard JSON, then `pick --workflow ops` re-ranks without re-running any model. - Extensible: a new language is one Lang entry; a new workflow is one block in workflows.json. Also fix a --runs grant-persistence bug in bench-sandbox.py: the grant leaked across runs, invalidating the L0-nogrant refusal test on runs 2+. Each run now revokes the ACL and starts ungranted. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
32 lines
1.2 KiB
Python
32 lines
1.2 KiB
Python
#!/usr/bin/env python3
|
|
"""bench-lang.py — launcher for the multi-language capability benchmark + picker.
|
|
|
|
This is the third hack-house benchmark, complementing:
|
|
• bench-ai.py — /ai chat latency/throughput on the real relay path
|
|
• bench-sandbox.py — /ai !task sandbox code-execution + safety guards
|
|
|
|
bench-lang answers the capability question MultiPL-E was built for: *can this
|
|
model actually write correct code in my language?* across Python, JavaScript,
|
|
Go, Rust and Bash — then weights the result by your workflow to recommend a
|
|
model. The implementation lives in the `bench/` package next to this file.
|
|
|
|
Examples:
|
|
.venv/bin/python hh/scripts/bench-lang.py langs
|
|
.venv/bin/python hh/scripts/bench-lang.py run \
|
|
--models qwen2.5-coder:3b qwen2.5:3b --languages python bash --limit 10
|
|
.venv/bin/python hh/scripts/bench-lang.py pick --workflow ops
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import sys
|
|
from pathlib import Path
|
|
|
|
# Make the sibling `bench/` package importable when run as a plain script.
|
|
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
|
|
|
from bench.cli import main # noqa: E402
|
|
|
|
if __name__ == "__main__":
|
|
sys.exit(main())
|