leetcrypt
7fb3911550
Add bench-lang.py + bench/ package: a third benchmark axis answering "which open-source model is best for my workflow?" across Python, JavaScript, Go, Rust and Bash. - MultiPL-E (Go/Rust/JS/Bash) + original HumanEval (Python), loaded via the HF datasets-server REST API with on-disk cache — no datasets/ pyarrow dependency. - Completions go straight to Ollama /api/generate with raw=True so instruct models continue the code instead of replying with prose. - Code runs in rootless, network-less podman (safe default) with a host-toolchain fallback; pass@1/pass@k via the HumanEval estimator. - run/score separation: results persist to a scorecard JSON, then `pick --workflow ops` re-ranks without re-running any model. - Extensible: a new language is one Lang entry; a new workflow is one block in workflows.json. Also fix a --runs grant-persistence bug in bench-sandbox.py: the grant leaked across runs, invalidating the L0-nogrant refusal test on runs 2+. Each run now revokes the ACL and starts ungranted. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
19 lines
908 B
Python
19 lines
908 B
Python
"""hh model-benchmark toolkit.
|
|
|
|
A small, extensible harness for answering one question: *which open-source model
|
|
works best for my workflow?* It has two axes, kept deliberately separate:
|
|
|
|
• capability-per-language — can the model write correct Go/Rust/Python/Bash/JS?
|
|
(driven by MultiPL-E + the original HumanEval, executed in a sandbox)
|
|
• tool-path fitness — does the model behave on hack-house's own /ai chat and
|
|
!task sandbox paths? (the existing bench-ai.py / bench-sandbox.py harnesses)
|
|
|
|
Both feed a common scorecard (score.py), which a workflow profile then weights
|
|
into a single ranked recommendation. Everything is dependency-light: model
|
|
completions go straight to Ollama's HTTP API, datasets come from the Hugging
|
|
Face datasets-server REST endpoint (no `datasets`/`pyarrow` install), and code
|
|
runs in rootless podman (with a host-toolchain fallback).
|
|
"""
|
|
|
|
__version__ = "0.1.0"
|