hack-house/hh/scripts/bench/olympics/SPEC.md

# Agent Olympics — a hackathon-competition benchmark inside hack-house

> Status: **DRAFT spec** (design only — no implementation yet). Author: bench team.
> Scope: a fourth benchmark axis layered on `hh/scripts/bench/`. Arena-mode first,
> MBPP/MultiPL-E bootstrap, modular team composition, Claude/Opus LLM-as-judge.
> This document is the build contract. It is grounded in the local Obsidian vault
> (`~/coding/obsidian/research/`) and 2026 web research — see **References**.

---

## 1. One-paragraph concept

Stand up **two (or more) teams of LLM agents** inside hack-house. Each team lives in
its own end-to-end-encrypted room and believes it has a private, secure channel to
its teammate(s). Each team is given the *same* coding challenge and an isolated VM
to work in. Teams **deliberate in chat**, then **implement and test code in their
VM**, racing on a blend of correctness, speed, code quality, and collaboration.
A referee posts challenges and records everything; a judge (deterministic tests +
Claude/Opus LLM-as-judge) scores the submissions and the conversation/tool-call
trajectory. Aggregate across a slate of events into a **medal table**. The framework
is the apparatus; the output is a reproducible answer to *"which model(s), in which
team configuration, collaborate best to ship good code fast?"*

This doubles as the most demanding live test of hack-house itself (concurrent
agents, encrypted rooms, shared sandbox, ACL, the destructive guard).

---

## 2. Goals & non-goals

### Goals
- **G1 — Real comms substrate.** Team deliberation flows through real hack-house
  E2E-encrypted rooms, not a simulated bus. The product is the medium.
- **G2 — Modular team composition.** Same-model teams (collaboration-protocol study)
  AND mixed-model teams (capability ladder) are first-class, config-driven.
- **G3 — Bootstrap fast.** Reuse the existing `bench/` MBPP + MultiPL-E loaders,
  graders, and `PodmanRuntime` so M1 runs end-to-end with zero new datasets.
- **G4 — Extensible challenges.** A `Challenge` interface so custom multi-file
  "file-type" challenges plug in later without touching the arena core.
- **G5 — Trustworthy scoring.** Verifiable (unit-test) scoring first; LLM-as-judge
  only for open-ended quality, with documented bias mitigation and judge calibration.
- **G6 — Reproducible & auditable.** Every frame, tool call, and VM state diff is
  recorded to a replayable transcript; runs are seed/version-pinned.
- **G7 — Measure the placebo.** Treat competitive/persona "performance elicitation"
  framing as an explicit, A/B-testable variable, not folklore (see §14).
- **G8 — Safe by construction.** Isolated VMs, deny-by-default egress, reuse the
  `bench/safety` guard + injection classifier as a fair-play/safety referee.

### Non-goals (for now)
- Not editing `cmd_chat/` (tool code). Arena mode drives turns externally and posts
  into real rooms. Product mode (real `AgentBridge` agents talking to each other)
  is a later milestone that needs an explicit greenlight.
- Not a training loop (no RL/fine-tuning). This selects and compares; it does not train.
- Not a public leaderboard. Held-out, private challenge sets are a feature, not a gap
  (contamination resistance — see §10).

---

## 3. Prior art this design borrows from (grounding)

- **Read-vs-write rule for multi-agent work.** Coding is *write-heavy, shared-state*
  work, the regime where naive parallel multi-agent systems fail (Cognition's
  "Flappy Bird" conflicting-implicit-decisions failure). The fix: **separate a
  read/deliberate phase from a single-driver write phase**, and **share full traces,
  not just messages**. We bake this into the loop (§7) *and* measure the coordination
  tax. [[vault: multi-agent-orchestration-patterns]]
- **Coordination topologies & milestone KPIs.** MultiAgentBench/MARBLE (ACL 2025)
  scores collaboration with milestone-based KPIs across star/chain/tree/graph
  topologies; AgentCoder/AgileCoder show role specialization (planner/coder/tester)
  beats undifferentiated peers. We make role + topology config knobs. [web]
- **Eval-harness discipline.** Standardize the harness, own a private held-out set;
  pick challenges for **discrimination, not difficulty**; report **intervals, not
  point estimates**; prefer verifiable scoring, reserve judges for open-ended quality.
  [[vault: eval-harnesses-benchmark-design]]
- **Outcome vs trajectory.** Final-output-only scoring overstates quality 20–40%;
  score the *path* (tool correctness, step efficiency, plan adherence, contribution
  balance). Use **agent-as-a-judge** to walk the trajectory. Report **pass^k**
  (worst-case) beside pass@k for reliability. [[vault: agent-evaluation-and-observability]]
- **Sandbox isolation hierarchy.** Plain containers are the *floor* (shared kernel =
  one-CVE-from-escape); microVMs (Firecracker/Kata) or gVisor are the bar for
  untrusted multi-tenant code; deny-by-default egress + hard caps + disposable
  one-shot. Current hh uses `podman --network=none` (the container floor) — we note
  the upgrade path. [[vault: agent-sandboxing-isolation]]
- **LLM-as-judge biases.** Position, verbosity, self-preference, format, calibration
  drift are all documented and individually mitigable; calibrate the judge against a
  human-labeled gold slice before trusting it. [web + [[vault: llm-evals]]]

---

## 4. Design pillars (constraints every component obeys)

1. **The room is the bus.** All inter-agent messages are real encrypted-room frames.
   The referee is a privileged room member (holds the key) for recording.
2. **Arena now, product later.** Orchestrator schedules turns + drives inference, but
   *posts every utterance into the real room*. Graduate to real `AgentBridge` agents
   in M5.
3. **Phase-separated collaboration.** Deliberate (read/plan, parallel-friendly) →
   Implement (single driver writes to the VM) → Test/iterate → Submit. This is the
   research-backed shape for write-heavy work and also the thing we measure.
4. **Everything modular & data-driven.** Teams, roles, models, topologies, challenges,
   scoring weights, and elicitation framing are config, not code.
5. **Verifiable-first scoring.** Hidden unit tests gate the score; judges only grade
   what can't be checked mechanically.
6. **Cost is a first-class metric.** Multi-agent ≈ 15× chat tokens; track tokens,
   wall-clock, turns. Report cost-normalized scores so a win bought with 10× spend
   is visible.
7. **Disposable, isolated, deny-by-default.** One VM per team per event, no host
   secrets mounted, egress blocked by default, hard resource caps.
8. **Reproducible.** Pin model versions, seeds, prompts, challenge set hash, and
   judge version into every result record.

---

## 5. Architecture

### 5.1 Package layout
```
bench/olympics/
  SPEC.md            # this document
  arena.py           # orchestrator: rooms, turn scheduler, phase machine, termination
  team.py            # Team, Member: persona/role/model binding, topology
  roster.py          # build teams from config (same-model / mixed-model, modular)
  challenge.py       # Challenge ABC + grader interface; MBPP/MultiPL-E adapters
  challenges/        # event specs (bootstrap: pointers into existing bench suites)
  vm.py              # per-team isolated workspace (wraps bench/runtime.py; egress policy)
  loop.py            # the collaboration phase machine (deliberate/implement/test/submit)
  comms.py           # hack-house room client for the arena (connect, post, record)
  transcript.py      # OTel-aligned event log -> replayable JSON per team/event
  scoring.py         # composite score, normalization, intervals, medal table
  judge/
    __init__.py
    deterministic.py # unit-test/lint/complexity scoring (no model)
    llm_judge.py     # Claude/Opus judge client + bias-mitigation harness
    rubric.py        # criterion-separated rubrics per axis
    prompts/         # judge system prompts (versioned)
  referee.py         # fair-play + safety: reuse bench/safety (guard + injection)
  elicitation.py     # persona/framing variants for the placebo experiment (§14)
bench-olympics.py    # launcher: run / replay / score / medal / judge subcommands
```

### 5.2 Reuse map (what already exists in `bench/`)
| Need | Existing component |
|------|--------------------|
| Problems + hidden tests | `bench/suites.py` (HumanEval/MBPP), `bench/datasets.py`, `bench/langs.py` |
| Pass@k / grading | `bench/harness.py` (`_pass_at_k`, assemble+run) |
| Isolated execution | `bench/runtime.py` (`PodmanRuntime`, `--network=none`, caps) |
| Model inference | `bench/completion.py` (raw) + new chat client in `comms.py` |
| Workflow weighting | `bench/score.py` + `workflows.json` pattern (reused for scoring profiles) |
| Safety referee | `bench/safety/` (`DESTRUCTIVE`, `classify.py`, `inject_bench.py`) |

### 5.3 Data flow (one event, one team)
```
config ─► roster.build_teams ─► arena.run_event
                                   │
   referee posts Challenge brief ──┼──► (room frame, recorded)
                                   ▼
        ┌──────────── loop (phase machine) ────────────┐
        │  DELIBERATE: members post plan/critique turns │  ◄─ inference via comms/judge model
        │  IMPLEMENT : driver !task → commands → vm.run │  ◄─ PodmanRuntime, egress policy
        │  TEST      : run PUBLIC tests in vm, feedback │
        │  (iterate until green or budget exhausted)    │
        │  SUBMIT    : freeze vm artifact               │
        └───────────────────────────────────────────────┘
                                   ▼
   transcript.json + frozen VM artifact ─► judge (deterministic + LLM) ─► scoring ─► medal table
```

### 5.4 hack-house integration points
- **Rooms:** one room per team (isolation). Arena connects as N agent clients
  (one per member) + 1 referee client, mirroring how `bench-sandbox.py` already
  drives `Client`/WebSocket sessions.
- **Sandbox:** the team's `!task` path types commands into the shared PTY; `vm.py`
  wraps `PodmanRuntime` for the actual isolated execution + output capture.
- **ACL:** the referee acts as room owner, issuing `_perm:acl` to grant the driver
  `drivers` rights for the Implement phase only; revoked between phases.
- **Guard/HITL:** the existing `DESTRUCTIVE` gate stays live; the referee can require
  host sign-off (the host-sign-off gate, separately specced) for flagged plans.

---

## 6. The collaboration loop (phase machine)

State machine per (team, event), bounded by a shared **budget**
(`max_rounds`, `max_tokens`, `wall_clock_s` — whichever trips first):

1. **BRIEF.** Referee posts the challenge to the room: task prompt, **public**
   example I/O, allowed languages, budget, and the submission protocol.
2. **DELIBERATE** (read/plan; parallel-friendly). Round-robin over members; each sees
   the full shared room trace (Cognition: *share full traces, not just messages*) and
   posts one message — proposal, critique, interface decision. Ends on a consensus
   token (e.g. `PLAN-LOCKED`) or round cap.
3. **IMPLEMENT** (write; single driver). The role-designated driver translates the
   locked plan into shell/file commands via `!task`; teammate(s) may post review
   comments but only the driver writes to the VM. (This is the research-backed way to
   avoid conflicting implicit decisions in write-heavy work.)
4. **TEST & ITERATE.** Run **public** tests in the VM; failures return to the room as
   feedback; loop DELIBERATE↔IMPLEMENT until green or budget exhausted.
5. **SUBMIT.** Team emits `SUBMIT`; VM artifact frozen and graded on **hidden** tests.

**Termination:** `SUBMIT`, budget exhaustion, or **no-progress** detection
(no new code + repeated/semantically-duplicate messages over a window).
**Topology knob:** DELIBERATE supports star (lead routes), chain, or free mesh — the
collaboration pattern itself becomes an experimental variable.

---

## 7. Teams, roles, personas (modular)

```yaml
# example team config
teams:
  - id: falcon
    topology: star          # star | chain | mesh
    members:
      - name: archie
        model: qwen2.5-coder:7b
        role: architect      # decomposes, sets interfaces, reviews, drives plan
        persona: senior-systems-engineer
      - name: bob
        model: qwen2.5-coder:7b
        role: builder         # implements; the Implement-phase driver
        persona: fast-prototyper
  - id: kestrel
    topology: mesh
    members:                  # mixed-model team
      - { name: kira,  model: qwen2.5:3b,        role: peer, persona: pragmatist }
      - { name: kojo,  model: llama3.2:3b,       role: peer, persona: skeptic }
```

- **Roles** map to persona system prompts + loop privileges (who drives Implement,
  who must approve `PLAN-LOCKED`). Built-ins: `architect`, `builder`, `tester`,
  `peer`. Role specialization is supported because prior art shows it helps; pure-peer
  teams are the control.
- **Modularity requirements:** same model on all members (protocol study), distinct
  models per member (capability study), distinct models per *team* (model-vs-model),
  and N-member teams (default 2; ≥3 allowed). All from config, no code change.
- **Personas** live in `elicitation.py` as named, versioned prompt fragments so the
  placebo experiment (§14) can swap them while holding model/challenge fixed.

---

## 8. Challenge system

`Challenge` is an ABC the arena consumes; graders are pluggable.

```python
class Challenge(Protocol):
    id: str
    languages: list[str]
    def brief(self) -> str: ...                  # prompt + PUBLIC examples (room-posted)
    def scaffold(self, vm) -> None: ...          # seed files into the VM (optional)
    def public_tests(self) -> Test: ...          # visible to the team during TEST
    def hidden_tests(self) -> Test: ...          # held out; used only at SUBMIT grading
    def rubric(self) -> Rubric: ...              # open-ended quality criteria for the judge
```

- **Bootstrap (M1–M3): MBPP / MultiPL-E adapter.** Wrap existing `bench/suites.py`
  problems: the MultiPL-E/MBPP `prompt` + visible examples become `brief()`/
  `public_tests()`, and a held-out slice of the asserts becomes `hidden_tests()`
  (public/hidden split prevents teaching-to-the-test). Languages from `bench/langs.py`.
- **Events = challenge archetypes (the "Olympics"):**
  | Event | Shape | Primary metric |
  |-------|-------|----------------|
  | Sprint | one easy problem | speed (turns + wall-clock) |
  | Marathon | hard / multi-file build | correctness (hidden pass@k) |
  | Relay | disjoint modules per member | interface-handshake success |
  | Debugging | fix a broken repo (injected bugs) | time-to-green |
  | Code review | catch a planted bug | collaboration / detection |
  | Security | resist a socially-engineered unsafe ask | resistance (reuse §13) |
- **Custom "file-type" challenges (post-bootstrap, G4):** multi-file projects with a
  scaffold + a containerized test command. Author once as a `challenges/<id>/` dir
  (brief.md, scaffold/, public_tests/, hidden_tests/, rubric.json) — the arena needs
  no changes. **Pick for discrimination:** retire any event all teams ace or all fail.

---

## 9. Scoring model

Composite per (team, event); weights are a profile (same mechanism as `workflows.json`).

```
event_score = wc·correctness + ws·speed + wq·quality + wb·collaboration + (penalties)
```

| Axis | Source | How |
|------|--------|-----|
| **Correctness** | deterministic | hidden-test **pass@k**; gate: 0 here caps the rest. Also report **pass^k** (worst-case reliability across repeated runs). |
| **Speed** | deterministic | normalized turns-to-green + wall-clock; tie-break tokens. |
| **Quality** | deterministic + judge | lint + cyclomatic complexity (deterministic) and LLM-judged readability/design against the rubric. |
| **Collaboration** | trajectory + judge | contribution balance (message/edit distribution), plan adherence, did review catch a bug, redundant-step rate. Agent-as-judge walks the transcript. |
| **Penalties** | referee | destructive/injection events, budget overrun, no-progress stalls. |

- **Normalization & intervals.** Per-event z-score or min-max across teams; **report
  confidence intervals** (multiple seeds / problem samples) and **do not rank teams
  whose intervals overlap** (vault eval discipline).
- **Cost-normalized variant.** Also publish `score / tokens` and `score / wall-clock`
  so a 15×-spend win is not mistaken for a free one.
- **Medal table.** Aggregate event_scores into per-team standings across the slate
  (gold/silver/bronze per event + overall), with weights per `olympics-profile`.

---

## 10. Judging (deterministic + Claude/Opus LLM-as-judge)

Two layers; LLM-judge only where deterministic checks can't reach.

### 10.1 Deterministic (always-on, free of model bias)
Unit-test pass@k/pass^k, lint, complexity, build success — in `judge/deterministic.py`.

### 10.2 LLM-as-judge — three operating modes (per user requirement)
The judge can **orchestrate** (drive a run live) or **analyze** (post-hoc), at two scales:

1. **In-session judge.** Claude in the *current* session reads one event's transcript
   + VM diff + tool-call log and scores the open-ended axes. Fast, interactive,
   good for a single event or while iterating on the framework.
2. **Skill judge — individual.** A `/olympics-judge` skill (one Skill invocation =
   one isolated judging session) pinned to **Opus 4.x (`claude-opus-4-8`)** grades one
   submission package. Clean context per submission → no cross-contamination; good for
   careful single-event grading at higher capability than the competitors.
3. **Skill judge — batch / large-scale.** The launcher fans out many `/olympics-judge`
   sessions (one per submission) for a full tournament, then aggregates. Parallel,
   reproducible, scales to many teams × events.

**Judge input package** (what every judge mode receives):
- the frozen **VM submission** (final file tree + build/test output),
- the full **hack-house conversation log** for the team (recorded transcript),
- the **tool-call / command log** (every `!task` → commands → result),
- the **final VM state** (diff vs scaffold), and
- the **rubric** (criterion-separated) for the event.

**Bias mitigation (mandatory, from the research):**
- **Position bias:** when comparing two teams pairwise, randomize/swap order and
  aggregate; prefer **independent rubric scoring** then derive A-vs-B from scores.
- **Verbosity bias:** rubric scores quality per-criterion, not "which is longer."
- **Self-preference:** never let a competitor model judge its own family; the judge
  (Opus 4.x) is stronger than and distinct from the local competitors.
- **Calibration:** validate the judge against a small human-labeled gold slice and
  report judge↔human agreement before trusting judge scores for ranking; pin
  judge model version + prompt version in results.

---

## 11. Observability & transcript schema

Record everything as spans aligned to the **OpenTelemetry GenAI semantic conventions**
(`invoke_agent`, `execute_tool`, `gen_ai.client.token.usage`) so traces are portable
and the judge gets clean structured input.

```jsonc
// transcript event (one per room frame / model call / tool call / phase change)
{
  "ts": 0.0, "event_id": "...", "team": "falcon", "challenge": "mbpp-py-42",
  "phase": "IMPLEMENT", "kind": "execute_tool",     // message|invoke_agent|execute_tool|phase|acl|guard
  "actor": "bob", "role": "builder",
  "model": "qwen2.5-coder:7b", "model_version": "...",
  "payload": { "task": "...", "commands": ["..."], "rc": 0, "out_trunc": "..." },
  "tokens": { "in": 0, "out": 0 }, "redactions": []
}
```
- One transcript JSON per (team, event); a run manifest pins config hash, seeds,
  model versions, challenge-set hash, judge version → full reproducibility.
- `bench-olympics.py replay <transcript>` re-renders the room conversation for audit;
  re-judging under a new rubric is a pure function of the recorded package.

---

## 12. Isolation, security & fair play

- **VM per team per event**, disposable/one-shot, **no host secrets mounted**,
  **egress deny-by-default**. M1 uses the existing `PodmanRuntime --network=none`
  (the "container floor"). **Upgrade path** for stronger/multi-tenant isolation:
  gVisor, then Kata/Firecracker microVMs, or a hosted sandbox (E2B/Modal) — sized to
  the threat (vault sandboxing hierarchy).
- **Team isolation** via separate encrypted rooms + separate VMs; no cross-room keys.
  No internet means teams can't fetch a published solution.
- **Fair-play referee** (`referee.py`) reuses `bench/safety`: the `DESTRUCTIVE` guard
  flags dangerous commands; the injection classifier (`bench/safety/classify.py`)
  watches for sabotage / social-engineering. Violations → penalties + logged.
- **Equal budget** enforced identically per team (rounds/tokens/wall-clock).

---

## 13. Performance elicitation — the "placebo" experiment (G7)

Hypothesis (user's): the *framing* — a high-stakes secure competition with elite
personas and a private teammate channel — pushes models toward better, more novel
solutions. The literature is mixed (role prompting and stakes/emotional framing show
real but inconsistent effects), so we **make framing a measured variable**, not an
assumption.

- **Treatment knobs** (in `elicitation.py`, swappable while model+challenge fixed):
  - *Competition framing*: "you are competing against another team; the faster,
    cleaner solution wins" vs. neutral "solve this task."
  - *Persona priming*: "elite staff engineer known for elegant, novel solutions" vs.
    bare assistant.
  - *Stakes/audience*: visible scoreboard, time pressure, "your teammate is counting
    on you" vs. none.
  - *Secure-comms theater*: explicitly tell agents the channel is private/encrypted
    (true) vs. silent.
- **Design:** A/B (factorial) — run identical models/challenges with framing ON vs
  OFF; compare correctness, quality, novelty, and **solution diversity** (distinct
  passing approaches). Report effect size with intervals; a knob only "works" if it
  beats the neutral control outside the confidence band.
- **Novelty metric:** cluster passing solutions (AST / embedding distance); reward
  approaches that pass hidden tests *and* differ from the canonical/most-common
  solution — this is where "novel solutions" become measurable rather than vibes.

---

## 14. Configuration & CLI

```
bench-olympics.py run     --config events/round1.yaml [--seed N] [--judge none|insession|skill]
bench-olympics.py replay  <transcript.json>
bench-olympics.py score   --run <run_dir> --profile balanced   # re-rank, no re-run
bench-olympics.py medal   --run <run_dir>
bench-olympics.py judge   --run <run_dir> --mode skill --model claude-opus-4-8 [--batch]
```
Config carries: teams (§7), event slate (§8), budget, scoring profile (§9),
elicitation arms (§13), runtime/isolation tier (§12), judge mode (§10).
Mirror the run/score separation already proven in `bench/score.py`: results persist,
`score`/`medal` re-rank without re-running a single model.

---

## 15. Build milestones (with acceptance criteria)

- **M1 — Arena spine.** One team (2 members, same model) solves one MBPP problem
  end-to-end: real room deliberation → driver `!task` → `PodmanRuntime` → public
  tests → SUBMIT → hidden-test grade → transcript.json.
  *Done when:* a full transcript replays and a deterministic score is produced.
- **M2 — Two teams, isolation, scoring.** Parallel rooms + VMs, composite score with
  intervals, cost tracking, scoreboard. *Done when:* two teams race the same event and
  a ranked result with CIs is emitted.
- **M3 — Events, roles, personas, topologies.** Event catalog (Sprint/Marathon/Relay/
  Debugging/Review), role-based loop privileges, medal table. *Done when:* a 3-event
  slate produces a medal table from config alone.
- **M4 — Judging + referee.** Deterministic quality + LLM-judge (in-session, then
  skill) with bias mitigation and a gold-slice calibration report; safety/fair-play
  referee live. *Done when:* judge↔human agreement is reported and penalties fire.
- **M5 — Product mode.** Real `AgentBridge` agents converse agent-to-agent (needs the
  bridge greenlight); arena nudges turns. *Done when:* an event completes using real
  product agents end-to-end.
- **M6 — Placebo experiment.** Factorial elicitation arms + novelty/diversity metrics.
  *Done when:* an A/B run reports framing effect sizes with intervals.

---

## 16. Open questions / decisions needed

1. **Turn-taking in Arena mode:** strict round-robin vs. a lightweight "who speaks
   next" router (star topology). Start round-robin (deterministic), add router in M3?
2. **Public/hidden split for MBPP:** how many asserts to reveal vs. hold out so
   `public_tests` guide without leaking the full spec? (Proposal: reveal 1 example,
   hold the rest.)
3. **Budget defaults:** rounds/tokens/wall-clock caps that keep an event under a few
   minutes locally while leaving room to actually collaborate.
4. **Judge model pinning:** confirm the exact Opus id for the skill judge
   (`claude-opus-4-8`) and whether batch judging runs via the Skill tool or a
   separate headless session.
5. **Isolation tier:** stay on `podman --network=none` for local runs, or invest in
   gVisor/microVM now for stronger guarantees and future multi-tenant use?
6. **Novelty metric:** AST-distance vs embedding-distance for solution diversity —
   which is cheap and discriminating enough locally?

---

## References

**Local vault** (`~/coding/obsidian/research/`):
- `2026-06-07-multi-agent-orchestration-patterns.md` (read-vs-write, share full traces, topologies, 15× cost)
- `2026-06-09-eval-harnesses-benchmark-design.md` (discrimination, intervals, verifiable-first, judge de-biasing, held-out sets)
- `2026-06-07-agent-evaluation-and-observability.md` (outcome vs trajectory, pass^k, agent-as-judge, OTel GenAI semconv)
- `2026-06-09-agent-sandboxing-isolation.md` (container floor → gVisor → microVM; deny-by-default egress; disposable VMs)
- `2026-06-09-securing-multi-agent-systems.md`, `2026-06-16-shared-memory-in-multi-agent-systems.md`, `2026-06-02-llm-evals.md`, `2026-06-07-agent-reliability-guardrails-and-hitl.md` (siblings)

**Web (2026):**
- MultiAgentBench / MARBLE — collaboration+competition KPIs, topologies — https://arxiv.org/abs/2503.01935 · https://github.com/ulab-uiuc/MARBLE
- AgentCoder / AgileCoder — role specialization in coding multi-agent — (see MultiAgentBench survey refs)
- LLM-as-Judge best practices & bias mitigation (2026) — https://futureagi.com/blog/llm-as-judge-best-practices-2026 · https://futureagi.com/blog/evaluating-llm-judge-bias-mitigation-2026/
- Judging LLM-as-a-Judge (MT-Bench biases) — https://arxiv.org/abs/2306.05685
- Beyond pass@1 (reliability / pass^k) — https://arxiv.org/pdf/2603.29231
- OpenTelemetry GenAI agent spans — https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-agent-spans/