leetcrypt
df8f1881d8
Fourth benchmark axis: teams of LLM agents deliberate in a room, implement code in an isolated VM, and are scored deterministically on correctness/speed. - Multi-language adapter (python/js/go/rust/bash) via MultiPL-E continuation mode - Append-only JSONL ledger with status tracking (ok/dnf/killed/error) so budget-exhausted or crashed runs still record a row (fixes selection bias) - Model-aware wall-clock scaling (U-shaped by param count; 3x for reasoning) - Self-owned SIGALRM/SIGTERM watchdog (RunTimeout: BaseException so broad except Exception handlers in the infer/completion path can't swallow it) - Seed forwarded to Ollama sampler + markdown-fence stripping in completion.py
10 lines
528 B
Python
10 lines
528 B
Python
"""Agent Olympics — a hackathon-competition benchmark inside hack-house.
|
|
|
|
This package layers a fourth benchmark axis on top of ``bench/``: teams of LLM
|
|
agents deliberate in a real hack-house room, implement code in an isolated VM,
|
|
and are scored on correctness/speed/quality/collaboration. See SPEC.md for the
|
|
full design. M1 is the arena spine (one same-model team, one MBPP problem,
|
|
real-room/local-bus deliberation -> driver implement -> PodmanRuntime -> hidden
|
|
tests -> replayable transcript -> deterministic score).
|
|
"""
|