feat(olympics): agent Olympics benchmark — multi-language arena, results ledger, model-aware budgets
Fourth benchmark axis: teams of LLM agents deliberate in a room, implement code in an isolated VM, and are scored deterministically on correctness/speed. - Multi-language adapter (python/js/go/rust/bash) via MultiPL-E continuation mode - Append-only JSONL ledger with status tracking (ok/dnf/killed/error) so budget-exhausted or crashed runs still record a row (fixes selection bias) - Model-aware wall-clock scaling (U-shaped by param count; 3x for reasoning) - Self-owned SIGALRM/SIGTERM watchdog (RunTimeout: BaseException so broad except Exception handlers in the infer/completion path can't swallow it) - Seed forwarded to Ollama sampler + markdown-fence stripping in completion.py
This commit is contained in:
@@ -0,0 +1,58 @@
|
||||
"""Team and Member value objects (config-driven, modular per SPEC §7).
|
||||
|
||||
A ``Member`` binds a room name to a model, a role (which selects persona +
|
||||
loop privileges) and an elicitation framing. A ``Team`` groups members under a
|
||||
topology. Same-model teams (protocol study) and mixed-model teams (capability
|
||||
study) are both just different config — no code changes.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
from . import elicitation
|
||||
|
||||
_DRIVER_ROLES = ("builder", "architect") # who may drive IMPLEMENT, in order
|
||||
|
||||
|
||||
@dataclass
|
||||
class Member:
|
||||
name: str
|
||||
model: str
|
||||
role: str = "peer"
|
||||
framing: str = "neutral"
|
||||
team: str = ""
|
||||
|
||||
def system_prompt(self) -> str:
|
||||
return elicitation.system_prompt(self.name, self.team, self.role,
|
||||
framing=self.framing)
|
||||
|
||||
|
||||
@dataclass
|
||||
class Team:
|
||||
id: str
|
||||
members: list[Member]
|
||||
topology: str = "star" # star | chain | mesh (M3 uses this)
|
||||
framing: str = "neutral"
|
||||
|
||||
def __post_init__(self):
|
||||
for m in self.members:
|
||||
if not m.team:
|
||||
m.team = self.id
|
||||
|
||||
@property
|
||||
def models(self) -> list[str]:
|
||||
return [m.model for m in self.members]
|
||||
|
||||
def driver(self) -> Member:
|
||||
"""The member who writes to the VM in IMPLEMENT. Prefer a builder, then
|
||||
an architect, else the first member."""
|
||||
for role in _DRIVER_ROLES:
|
||||
for m in self.members:
|
||||
if m.role == role:
|
||||
return m
|
||||
return self.members[0]
|
||||
|
||||
def speaking_order(self) -> list[Member]:
|
||||
"""Round-robin order for DELIBERATE (M1). M3 will route by topology."""
|
||||
return list(self.members)
|
||||
Reference in New Issue
Block a user