feat(olympics): agent Olympics benchmark — multi-language arena, results ledger, model-aware budgets
Fourth benchmark axis: teams of LLM agents deliberate in a room, implement code in an isolated VM, and are scored deterministically on correctness/speed. - Multi-language adapter (python/js/go/rust/bash) via MultiPL-E continuation mode - Append-only JSONL ledger with status tracking (ok/dnf/killed/error) so budget-exhausted or crashed runs still record a row (fixes selection bias) - Model-aware wall-clock scaling (U-shaped by param count; 3x for reasoning) - Self-owned SIGALRM/SIGTERM watchdog (RunTimeout: BaseException so broad except Exception handlers in the infer/completion path can't swallow it) - Seed forwarded to Ollama sampler + markdown-fence stripping in completion.py
This commit is contained in:
@@ -0,0 +1,9 @@
|
||||
"""Agent Olympics — a hackathon-competition benchmark inside hack-house.
|
||||
|
||||
This package layers a fourth benchmark axis on top of ``bench/``: teams of LLM
|
||||
agents deliberate in a real hack-house room, implement code in an isolated VM,
|
||||
and are scored on correctness/speed/quality/collaboration. See SPEC.md for the
|
||||
full design. M1 is the arena spine (one same-model team, one MBPP problem,
|
||||
real-room/local-bus deliberation -> driver implement -> PodmanRuntime -> hidden
|
||||
tests -> replayable transcript -> deterministic score).
|
||||
"""
|
||||
Reference in New Issue
Block a user