feat(olympics): agent Olympics benchmark — multi-language arena, results ledger, model-aware budgets

Fourth benchmark axis: teams of LLM agents deliberate in a room, implement code in an isolated VM, and are scored deterministically on correctness/speed. - Multi-language adapter (python/js/go/rust/bash) via MultiPL-E continuation mode - Append-only JSONL ledger with status tracking (ok/dnf/killed/error) so budget-exhausted or crashed runs still record a row (fixes selection bias) - Model-aware wall-clock scaling (U-shaped by param count; 3x for reasoning) - Self-owned SIGALRM/SIGTERM watchdog (RunTimeout: BaseException so broad except Exception handlers in the infer/completion path can't swallow it) - Seed forwarded to Ollama sampler + markdown-fence stripping in completion.py
2026-06-27 10:17:23 -07:00
parent 7fb3911550
commit df8f1881d8
19 changed files with 2962 additions and 2 deletions
@@ -0,0 +1,60 @@
+"""Personas and framing fragments — the building blocks of the §13 placebo
+experiment, used here only to assemble system prompts.
+
+A member's system prompt is composed of three swappable parts so the placebo
+A/B (M6) can vary framing while holding model + challenge fixed:
+
+  base      — role-neutral task instruction (always present)
+  persona   — who the model is told it is (role-flavoured)
+  framing   — competition/stakes/secure-comms theater (the treatment knob)
+
+M1 ships ``neutral`` framing as the control and ``competition`` as one treatment;
+the loop selects via config. Keeping these as named, versioned strings is what
+makes the elicitation effect *measurable* rather than hard-coded folklore.
+"""
+
+from __future__ import annotations
+
+# ── role personas ─────────────────────────────────────────────────────────
+PERSONAS: dict[str, str] = {
+    "architect": (
+        "You are the team's architect. You decompose the problem, fix the "
+        "interface and edge cases, and critique proposals crisply. You do not "
+        "write the final code yourself — you guide the builder."),
+    "builder": (
+        "You are the team's builder. You turn the locked plan into a single "
+        "correct, self-contained implementation. You favour simple, working "
+        "code over cleverness."),
+    "tester": (
+        "You are the team's tester. You hunt for failing inputs and edge cases "
+        "and report concrete bugs, not vague worries."),
+    "peer": (
+        "You are an engineer collaborating as an equal peer. You contribute "
+        "ideas, review your teammate's, and converge quickly on a plan."),
+}
+
+# ── framing arms (the placebo treatment) ──────────────────────────────────
+FRAMINGS: dict[str, str] = {
+    "neutral": (
+        "Work with your teammate to solve the coding challenge correctly."),
+    "competition": (
+        "This is a timed competition against another team over a private, "
+        "encrypted channel. The faster, cleaner, correct solution wins. Your "
+        "teammate is counting on you — be decisive and elegant."),
+}
+
+_BASE = (
+    "You are {name}, a member of team {team} in a collaborative coding event. "
+    "You communicate with your teammate(s) over a secure chat room. Keep each "
+    "message short and focused: propose, critique, or decide. When the team "
+    "agrees on an approach, say the token PLAN-LOCKED on its own line. Do not "
+    "write the full final solution in chat — that is the builder's job in the "
+    "implement phase.")
+
+
+def system_prompt(name: str, team: str, role: str, *,
+                  framing: str = "neutral") -> str:
+    parts = [_BASE.format(name=name, team=team),
+             PERSONAS.get(role, PERSONAS["peer"]),
+             FRAMINGS.get(framing, FRAMINGS["neutral"])]
+    return " ".join(parts)