feat(bench): multi-language capability benchmark + model picker

Add bench-lang.py + bench/ package: a third benchmark axis answering
"which open-source model is best for my workflow?" across Python,
JavaScript, Go, Rust and Bash.

- MultiPL-E (Go/Rust/JS/Bash) + original HumanEval (Python), loaded via
  the HF datasets-server REST API with on-disk cache — no datasets/
  pyarrow dependency.
- Completions go straight to Ollama /api/generate with raw=True so
  instruct models continue the code instead of replying with prose.
- Code runs in rootless, network-less podman (safe default) with a
  host-toolchain fallback; pass@1/pass@k via the HumanEval estimator.
- run/score separation: results persist to a scorecard JSON, then
  `pick --workflow ops` re-ranks without re-running any model.
- Extensible: a new language is one Lang entry; a new workflow is one
  block in workflows.json.

Also fix a --runs grant-persistence bug in bench-sandbox.py: the grant
leaked across runs, invalidating the L0-nogrant refusal test on runs 2+.
Each run now revokes the ACL and starts ungranted.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
leetcrypt
2026-06-17 09:39:27 -07:00
parent c5715ba2e3
commit 7fb3911550
11 changed files with 733 additions and 1 deletions
+9 -1
View File
@@ -97,6 +97,10 @@ class Owner(Client):
await self._send(ws, json.dumps(
{"_perm": "acl", "drivers": [agent], "sudoers": [agent] if sudo else []}))
async def revoke(self, ws) -> None:
await self._send(ws, json.dumps(
{"_perm": "acl", "drivers": [], "sudoers": []}))
async def task(self, ws, agent: str, task: str) -> None:
await self._send(ws, f"/ai {agent} !{task}")
@@ -334,9 +338,13 @@ async def run(args, agent_name: str) -> list[dict]:
"exec": "", "note": "", "passes": f"0/{args.runs}", "avg_s": 0.0}
for lvl in LEVELS]
granted = False
for r in range(args.runs):
tag = f"[run {r + 1}/{args.runs}] " if args.runs > 1 else ""
# Each run must start ungranted so the L0-nogrant refusal test is
# valid every time — otherwise run 1's grant leaks into runs 2+.
await owner.revoke(ws)
await asyncio.sleep(0.6)
granted = False
for lvl in LEVELS:
if lvl["phase"] == "granted" and not granted:
await owner.grant(ws, agent_name, sudo=args.sudo)