Fourth benchmark axis: teams of LLM agents deliberate in a room, implement
code in an isolated VM, and are scored deterministically on correctness/speed.
- Multi-language adapter (python/js/go/rust/bash) via MultiPL-E continuation mode
- Append-only JSONL ledger with status tracking (ok/dnf/killed/error) so
budget-exhausted or crashed runs still record a row (fixes selection bias)
- Model-aware wall-clock scaling (U-shaped by param count; 3x for reasoning)
- Self-owned SIGALRM/SIGTERM watchdog (RunTimeout: BaseException so broad
except Exception handlers in the infer/completion path can't swallow it)
- Seed forwarded to Ollama sampler + markdown-fence stripping in completion.py
Add bench-lang.py + bench/ package: a third benchmark axis answering
"which open-source model is best for my workflow?" across Python,
JavaScript, Go, Rust and Bash.
- MultiPL-E (Go/Rust/JS/Bash) + original HumanEval (Python), loaded via
the HF datasets-server REST API with on-disk cache — no datasets/
pyarrow dependency.
- Completions go straight to Ollama /api/generate with raw=True so
instruct models continue the code instead of replying with prose.
- Code runs in rootless, network-less podman (safe default) with a
host-toolchain fallback; pass@1/pass@k via the HumanEval estimator.
- run/score separation: results persist to a scorecard JSON, then
`pick --workflow ops` re-ranks without re-running any model.
- Extensible: a new language is one Lang entry; a new workflow is one
block in workflows.json.
Also fix a --runs grant-persistence bug in bench-sandbox.py: the grant
leaked across runs, invalidating the L0-nogrant refusal test on runs 2+.
Each run now revokes the ACL and starts ungranted.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
bench-ai.py drives the /ai chat path end-to-end (SRP -> Fernet -> WebSocket
-> provider), measuring TTFT/total/tok-s per model, with a --direct provider
mode that isolates raw model throughput from event-loop contention.
bench-sandbox.py benchmarks the /ai <agent> !<task> sandbox code path by
playing the room owner on the zero-knowledge relay: it grants drive, sends
graded tasks (L0 no-grant refusal, file/script/logic/multistep, destructive
gating+confirm, blast-radius cap), captures the agent's injected _sbx:input
frames, and grades correctness behind the same destructive guard. Includes
REPL-aware exec replay (folds python3 sessions into heredocs), prompt-prefix
stripping, model-fail vs replay-limit tagging, --runs N averaging with
per-step timing, and auto-bumped timeouts for reasoning models.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add a -h/--help guard (prints the usage header) to the five scripts
that lacked one: smoke.sh, smoke-e2e.sh, test-features.sh,
demo-save-load.sh, sandbox-bootstrap.sh. All 15 hh/scripts/*.sh now
respond to --help.
- README: new "Scripts" section detailing every script (setup, hosting,
sandbox provisioning, tests) and noting each takes --help; plus an
archive note for the film-* recording scripts.
- README: new "Window layout" section documenting live pane resizing
(F4 fullscreen, click/F5 to select + arrows to resize) and the
/layout save/load/list/rm/reset presets, with matching keybinding-table
rows.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Trim newcomer-facing clutter in hh/scripts/ without changing the running
product or CI.
- Move the one-off demo-recording scripts (film-save-load.sh,
film-virtualbox.sh) to scripts/archive/ — they depend on external
personal tooling (asciinema, video-toolkit, edge-tts, xdotool) and were
never wired into the README or CI.
- Delete join.sh: it duplicated connect.sh's join with fewer options
(hardcoded port, plaintext-only). Its only unique behaviour — a
pre-join git pull (gitea→origin, ff-only) + rebuild — is now an opt-in
`--sync` flag on connect.sh, leaving one canonical joiner.
No scripts or docs referenced the removed/moved files, so nothing else
needed updating.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previously, launching a sandbox on a machine without the backend binary
died with a raw "not installed" error and no path forward. Now the
missing backend can be installed in-line, gated on explicit consent, so
a fresh checkout can go from zero to a running sandbox without leaving
the TUI.
Folded into the existing `/sbx launch` verb rather than a new command
(menu is already dense): `/sbx launch docker|multipass [image] install`.
The `install` token opts in; without it the user gets an actionable
error naming the exact retry. The token is filtered out of positional
image parsing so it never shadows a custom image.
The install runs off-thread inside the existing spawn_launch task,
before provisioning, so the TUI never blocks and the *launching guard is
cleared via BrokerMsg on failure. A fresh Docker install also leaves its
daemon up, so launch proceeds straight through.
Scripts (detect-first, never silent, --plan dry-run, idempotent if
already present):
- ensure-multipass.sh (new): Linux→snap (clear failure if snapd absent),
macOS→brew cask, Windows→winget.
- ensure-docker.sh: new --install mode using Docker's OFFICIAL,
GPG-verified apt repo (docker-ce) on Debian/Ubuntu, with dnf
(Fedora/RHEL) and pacman (Arch) fallbacks. Deliberately avoids piping
get.docker.com into a root shell. Existing daemon-start path intact.
sbx.rs: docker_installed()/multipass_installed() detectors and
ensure_docker_install()/ensure_multipass_install() wrappers; ENSURE_MULTIPASS const.
ui.rs: help text documents the [install] option on both backends.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The existing vbox paths only open pre-made images; there was no way to
build one from scratch. Add `/sbx launch vbox new [name]` backed by
scripts/vbox-new.sh: download an Ubuntu cloud image, convert it to a VDI,
build a cloud-init NoCloud seed that creates a sudo login user and installs
the sandbox-tools.json toolchain on first boot, then create + boot the VM.
VirtualBox guests can't be exec'd into like docker/multipass, so cloud-init
is the provisioning channel. Generates a one-time login password (or takes
--pass) and authorizes a local SSH key if present. Runs off-thread; rolls
back a half-built VM on failure.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Reorganize the help menu into one VIRTUAL MACHINES cluster covering all
backends, and bring docker/multipass/vbox to save+load parity:
- Launch-time dev toolchain: sandbox-bootstrap.sh + editable
sandbox-tools.json (vim/curl guaranteed), installed in docker AND
multipass sandboxes at provision time.
- Vbox load: vm_restore + `/sbx vmload <vm> [label]` (restore snapshot
then boot the GUI).
- Multipass load: `/sbx load` is now backend-aware (locate_snapshot +
SnapKind), mp_restore re-attaches the shared shell; teardown stops
(not purges) an instance that still has snapshots so they survive
`/sbx stop`.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- app.rs: F1 now closes the help overlay (toggle), not just opens it
- host-house.sh: add --user/--name flag to override the $USER default seat
- connect.sh: incremental cargo build before launch so the UI is never a
stale release binary; --no-build escape hatch for toolchain-less joiners
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add a BRANCH guard (default: main) so launching the house never demos a
stale checkout. ensure_branch() switches to $BRANCH before the cargo build,
refuses when the tree is dirty (won't clobber uncommitted work), and no-ops
when already on-branch or when BRANCH= is passed to keep the current checkout.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- host-house.sh: one tmux session hosting a room (server window) plus your own
GUI client window — builds, frees the port, waits for /health, then attaches.
- host-room.sh: free_port() kills any stale LISTENer squatting on the port so
the bind can't fail with "address already in use".
- join.sh: cap each remote pull with `timeout` + GIT_TERMINAL_PROMPT=0 so an
unreachable remote fails fast instead of hanging.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move the 14 hack-house scripts (bootstrap, lets-hack, host-room, smoke,
demo/film harnesses, connect/join, ensure-docker/vbox) into hh/scripts/
and fix their path resolution for the deeper location. Update cross-refs:
sbx.rs script consts, app.rs hints, CI, direnv .envrc, README, and docs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
leetcrypt