Add bench-lang.py + bench/ package: a third benchmark axis answering
"which open-source model is best for my workflow?" across Python,
JavaScript, Go, Rust and Bash.
- MultiPL-E (Go/Rust/JS/Bash) + original HumanEval (Python), loaded via
the HF datasets-server REST API with on-disk cache — no datasets/
pyarrow dependency.
- Completions go straight to Ollama /api/generate with raw=True so
instruct models continue the code instead of replying with prose.
- Code runs in rootless, network-less podman (safe default) with a
host-toolchain fallback; pass@1/pass@k via the HumanEval estimator.
- run/score separation: results persist to a scorecard JSON, then
`pick --workflow ops` re-ranks without re-running any model.
- Extensible: a new language is one Lang entry; a new workflow is one
block in workflows.json.
Also fix a --runs grant-persistence bug in bench-sandbox.py: the grant
leaked across runs, invalidating the L0-nogrant refusal test on runs 2+.
Each run now revokes the ACL and starts ungranted.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
bench-ai.py drives the /ai chat path end-to-end (SRP -> Fernet -> WebSocket
-> provider), measuring TTFT/total/tok-s per model, with a --direct provider
mode that isolates raw model throughput from event-loop contention.
bench-sandbox.py benchmarks the /ai <agent> !<task> sandbox code path by
playing the room owner on the zero-knowledge relay: it grants drive, sends
graded tasks (L0 no-grant refusal, file/script/logic/multistep, destructive
gating+confirm, blast-radius cap), captures the agent's injected _sbx:input
frames, and grades correctness behind the same destructive guard. Includes
REPL-aware exec replay (folds python3 sessions into heredocs), prompt-prefix
stripping, model-fail vs replay-limit tagging, --runs N averaging with
per-step timing, and auto-bumped timeouts for reasoning models.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
leetcrypt