R7/RQ3: executor-wiring stone — offline selector DVs + calibration gate
Wire the RQ3 churn-resilience companion (severable follow-on; RQ3 frozen in its
own prereg) as SYNTHETIC/OFFLINE build + calibration only — no confirmatory
battery run.
- battery: enumerate_rq3_cells()/rq3_schedule() add selector in {static(control),
random, agent} at 1house/bridge-off under pinned churn kp=30/steps=20, kept
SEPARATE so the frozen 6-cell lead lattice stays byte-identical.
- executor: run_rq3_cell_run collects offline selector DVs (throughput-retention,
drops/rebuilds, rebuild-interval gaps); added-latency is live-only (offline path
records None, never fabricated); run_rq3_battery(live=False) hard-raises.
- analysis/rq3_calibration: DRY offline gate. Churn-bites (1589 drops/1576
rebuilds) + rebuild-classifier calibration green with real teeth — churned(kp30)
vs baseline(kp5) AUC 0.926 separable, baseline-vs-baseline null 0.518 blind.
Classifier scored on the PER-RUN mean inter-rebuild gap (the confirmatory
grouping unit); the frozen instrument (rebuild_interval_gaps,
rebuild_classifier_auc) is untouched, no fit to confirmatory data.
HARD HOLD: no RQ3 confirmatory battery (live added-latency is operator+grid-gated).
Lead prereg SHA f22331a72e… untouched; containment intact (synthetic/offline, $0
local Ollama, frontier arm inert); worktree-only.
Tests: tests/test_sor_rq3_wiring.py 9 passed; full SOR suite 194 passed.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -87,3 +87,7 @@ RQ2 RATIFIED (operator Andre, 2026-07-20, while blind) — observation-consisten
|
||||
- 2026-07-21 OVERSEER RULED (Andre): STOP + diagnosis correct; items 1–2 were WORDED under the naive-funnel prior, not broken. Re-worded `rq2p3-mechanism-prereg.md` §7 pre-freeze — item 1 re-anchored on the FROZEN `bridge-federated` branch (injective fresh-bridge → m_i=1 → H≈0, c=1/C; a pool draws with replacement so it can't/mustn't reproduce that); item 2 B=1 → c=1.0 AND high H (maximal mix, "low H" refuted by construction); added §7 scope note (gate validates instrument, must NOT pre-assert H-vs-c sign → the two-sided confirmatory question). Deviation logged in `docs/stage-05-rq2p3-gate-clarification.md` (no number invented, cites the dry-pass; H1/H2/H3 two-sided UNCHANGED — not HARKing, a pre-freeze validation-gate fix).
|
||||
- CALIBRATION RE-RAN GREEN (all 4 real teeth): item1 frozen branch labels-distinct+H_all_zero+c=1/C ✅; item2 c=1.0 & H=2.54 bits HIGH vs fresh-ref 0.0 ✅; item3 monotonicity ✅; item4 entropy=log2N ✅; all_pass=True. Pool anchor B=50 kept as EXPLORATORY mix diagnostic (ρ=+0.838), not a regression target. Tests: tests/test_sor_rq2p3_pool.py 9 passed. `confirm.py`/`stats.py`/`confirm_load_rq2.py` UNCHANGED.
|
||||
- FREEZE STAYS HUMAN GATE: §10 empty; NO confirmatory RQ2-P3 cell until Andre records SHA-256 + signs off. Honest disclosure logged for the paper: dry pass previews mix (ρ 0→+0.838) so the battery quantifies a dose-response already visible at calibration; two-sided pre-commitment stands. Lead prereg SHA f22331a72e… INTACT; worktree-only.
|
||||
- 2026-07-21 RQ3 EXECUTOR-WIRING STONE built + SYNTHETIC-tested (authorized severable follow-on; RQ3 already frozen in its own prereg). battery.enumerate_rq3_cells()/rq3_schedule() = selector∈{static(control),random,agent} at 1house/bridge-off under pinned churn kp=30/steps=20, kept SEPARATE (frozen 6-cell lead lattice byte-identical). executor.run_rq3_cell_run collects OFFLINE selector DVs (throughput-retention, drops/rebuilds, rebuild-interval gaps); added-latency stays live-only (offline path → None, never fabricated); run_rq3_battery(live=False) HARD-RAISES. analysis/rq3_calibration.py = DRY/offline §3-4 gate.
|
||||
- CHURN-BITES + CLASSIFIER GATES GREEN with real teeth (HARD HOLD satisfied): item1 churn bites — 1589 drops/1576 rebuilds, every cell fraction-with-drops=1.0; item2 rebuild-classifier AUC churned(kp30)-vs-baseline(kp5)=0.926≥0.90 SEPARABLE, baseline-vs-baseline null=0.518 blind; item3 agent cache-replay reproducible; item4 entropy=log2N. all_pass=True.
|
||||
- HARNESS FIX (not detector-tuning, logged): initial pooled-RAW-gap classifier landed AUC 0.779 (<0.90) — surfaced as candidate STOP, then diagnosed as a GROUPING-UNIT bug (pooling integer gaps floods AUC with ties). Switched calibration to PER-RUN mean-gap (the confirmatory grouping unit, cf. RQ2-P3) → 0.926 with a clean 0.518 null; frozen instrument (rebuild_interval_gaps, rebuild_classifier_auc) UNTOUCHED; rebuild-count rejected (teeth 1.0 but broken null 0.64). No fit to confirmatory data.
|
||||
- HARD HOLD REMAINS: NO RQ3 confirmatory battery run (run_rq3_battery live=True is operator+grid-gated; added-latency needs isolated-docker). Tests: tests/test_sor_rq3_wiring.py 9 passed; full SOR suite 194 passed (no regression). Lead prereg SHA f22331a72e… INTACT; RQ2-P3 §10 still empty; containment intact (synthetic/offline, $0 local); worktree-only.
|
||||
|
||||
@@ -0,0 +1,212 @@
|
||||
"""RQ3 companion instrument-validation gate (`rq3-companion-run-brief.md` §3-4).
|
||||
|
||||
DRY, SYNTHETIC-ONLY, OFFLINE calibration of the churn-resilient selector arm. It
|
||||
replays the pinned churn schedule under each selector strategy via the pure
|
||||
``run_selection`` (no engine, no traffic, no confirmatory record read) and reports
|
||||
the RQ3 GO gates that BLOCK the confirmatory battery:
|
||||
|
||||
1. **Churn-bites gate** (§4). At the pinned ``kill_prob_pct = 30``, ``steps = 20``
|
||||
the churn must actually bite — circuits genuinely lose hops and rebuild
|
||||
(non-zero drops/rebuilds). If the drop rate were trivially zero, the
|
||||
throughput-retention and rebuild-classifier tests would be degenerate → STOP.
|
||||
2. **Rebuild-classifier calibration gate** (§3.3). Calibrated on labelled control
|
||||
signals BEFORE the confirmatory cells: churned (``kp=30``) vs the LOW-CHURN
|
||||
baseline (``kp=5``) must be separable on the rebuild-interval-gap signal
|
||||
(AUC≈1), while baseline-vs-baseline must be indistinguishable (AUC≈0.5). Not
|
||||
fit to confirmatory data.
|
||||
3. **Agent-selector reproducibility** (§4). Same seed + churn history → byte-
|
||||
identical selector decisions (the deterministic local heuristic arm here; the
|
||||
Ollama confirmatory arm is reproducible via its committed decision-cache
|
||||
replay, exercised in the unit tests — no network in this gate).
|
||||
4. **Entropy calibration (inherited)** — Shannon plug-in of N equiprobable
|
||||
senders is exactly log2(N).
|
||||
|
||||
HARD HOLD: the confirmatory RQ3 battery does not run until items 1 AND 2 are green
|
||||
(brief §3-4); this module surfaces, it does not run the battery.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import statistics
|
||||
import sys
|
||||
from typing import Dict, List
|
||||
|
||||
from cmd_chat.sor.analysis.detectors import shannon_entropy_bits
|
||||
from cmd_chat.sor.analysis.metrics import rebuild_classifier_auc, throughput_retention
|
||||
from cmd_chat.sor.battery import derive_seed, enumerate_rq3_cells
|
||||
from cmd_chat.sor.churn import churn_schedule
|
||||
from cmd_chat.sor.executor import _rq3_pool, rebuild_interval_gaps
|
||||
from cmd_chat.sor.selector import run_selection
|
||||
|
||||
R_DRY = 30 # runs/regime for the dry calibration (matches §6 R)
|
||||
POOL_SIZE = 8 # 1-house consenting-node pool (> hops, with churn headroom)
|
||||
HOPS = 3
|
||||
CHURN_KP = 30 # pinned confirmatory churn (run-brief §2(B))
|
||||
CHURN_STEPS = 20 # pinned churn horizon
|
||||
BASELINE_KP = 5 # LOW-CHURN baseline for the classifier (run-brief §3.3)
|
||||
|
||||
|
||||
def _pool(size: int = POOL_SIZE) -> List[str]:
|
||||
return [f"1house/node{ix:02d}" for ix in range(size)]
|
||||
|
||||
|
||||
def _cal_seed(tag: str, i: int) -> int:
|
||||
return int.from_bytes(hashlib.sha256(f"rq3-cal|{tag}|{i}".encode()).digest()[:8], "big")
|
||||
|
||||
|
||||
def _replay(seed: int, kp: int, strategy: str = "static", *, steps: int = CHURN_STEPS,
|
||||
pool_size: int = POOL_SIZE, hops: int = HOPS):
|
||||
nodes = _pool(pool_size)
|
||||
sched = churn_schedule(seed, nodes, steps, kill_prob_pct=kp)
|
||||
return run_selection(seed, nodes, hops, sched, strategy=strategy)
|
||||
|
||||
|
||||
# --- Item 1: churn-bites ---------------------------------------------------- #
|
||||
def churn_bites_gate(r: int = R_DRY) -> Dict:
|
||||
"""At the pinned kp=30/steps=20, confirm the churn bites across every RQ3 cell:
|
||||
non-zero drops AND rebuilds, and a healthy fraction of runs that lose a hop."""
|
||||
per_cell: List[Dict] = []
|
||||
total_drops = 0
|
||||
total_rebuilds = 0
|
||||
for cell in enumerate_rq3_cells():
|
||||
kp = int(cell.factors["churn_kill_prob_pct"])
|
||||
steps = int(cell.factors["churn_steps"])
|
||||
strategy = cell.factors["selector"]
|
||||
drops = 0
|
||||
rebuilds = 0
|
||||
runs_with_drops = 0
|
||||
retentions: List[float] = []
|
||||
for ri in range(r):
|
||||
seed = derive_seed(cell.cell_id, ri)
|
||||
res = _replay(seed, kp, strategy=strategy, steps=steps)
|
||||
drops += res.drops
|
||||
rebuilds += len(res.rebuilds)
|
||||
runs_with_drops += 1 if res.drops > 0 else 0
|
||||
retentions.append(throughput_retention(res))
|
||||
total_drops += drops
|
||||
total_rebuilds += rebuilds
|
||||
per_cell.append({
|
||||
"cell_id": cell.cell_id,
|
||||
"strategy": strategy,
|
||||
"total_drops": drops,
|
||||
"total_rebuilds": rebuilds,
|
||||
"fraction_runs_with_drops": runs_with_drops / r,
|
||||
"mean_throughput_retention": statistics.fmean(retentions),
|
||||
})
|
||||
bites = all(c["total_drops"] > 0 and c["total_rebuilds"] > 0
|
||||
and c["fraction_runs_with_drops"] >= 0.5 for c in per_cell)
|
||||
return {
|
||||
"kill_prob_pct": CHURN_KP,
|
||||
"steps": CHURN_STEPS,
|
||||
"total_drops": total_drops,
|
||||
"total_rebuilds": total_rebuilds,
|
||||
"per_cell": per_cell,
|
||||
"pass": bool(total_drops > 0 and total_rebuilds > 0 and bites),
|
||||
}
|
||||
|
||||
|
||||
# --- Item 2: rebuild-classifier calibration --------------------------------- #
|
||||
def _per_run_gap_signal(tag: str, kp: int, r: int, strategy: str = "static") -> List[float]:
|
||||
"""Per-run rebuild-interval signal: the MEAN inter-rebuild gap for each run
|
||||
(the confirmatory grouping unit — cf. the RQ2-P3 gate, ``per-run``). Pooling
|
||||
raw integer gaps across runs would flood the AUC with tied low integers and
|
||||
conflate within- and between-run variation, depressing a genuine signal; per-
|
||||
run aggregation is the correct unit and matches how the confirmatory battery
|
||||
groups observations. The frozen instrument (``rebuild_interval_gaps``,
|
||||
``rebuild_classifier_auc``) is untouched — this is harness grouping only.
|
||||
Runs with <2 rebuilds yield no gap and are omitted (no fabricated interval)."""
|
||||
signal: List[float] = []
|
||||
for i in range(r):
|
||||
res = _replay(_cal_seed(tag, i), kp, strategy=strategy)
|
||||
gaps = rebuild_interval_gaps(res)
|
||||
if gaps:
|
||||
signal.append(statistics.fmean(gaps))
|
||||
return signal
|
||||
|
||||
|
||||
def rebuild_classifier_gate(r: int = R_DRY) -> Dict:
|
||||
"""Churned (kp=30) vs low-churn baseline (kp=5) must be separable on the per-run
|
||||
rebuild-interval-gap signal (AUC≈1); baseline-vs-baseline must not (AUC≈0.5).
|
||||
Calibrated on labelled control signals, never fit to confirmatory cells."""
|
||||
churned = _per_run_gap_signal("churned", CHURN_KP, r)
|
||||
baseline = _per_run_gap_signal("baseline", BASELINE_KP, r)
|
||||
# Two disjoint baseline halves for the null (same regime → indistinguishable).
|
||||
base_a = _per_run_gap_signal("baseline-A", BASELINE_KP, r)
|
||||
base_b = _per_run_gap_signal("baseline-B", BASELINE_KP, r)
|
||||
|
||||
auc_sep = rebuild_classifier_auc(churned, baseline)
|
||||
auc_null = rebuild_classifier_auc(base_a, base_b)
|
||||
separable = auc_sep >= 0.90
|
||||
null_ok = abs(auc_null - 0.5) <= 0.15
|
||||
return {
|
||||
"churned_kill_prob_pct": CHURN_KP,
|
||||
"baseline_kill_prob_pct": BASELINE_KP,
|
||||
"grouping_unit": "per-run mean inter-rebuild gap",
|
||||
"n_churned_runs": len(churned),
|
||||
"n_baseline_runs": len(baseline),
|
||||
"auc_churned_vs_baseline": auc_sep,
|
||||
"auc_baseline_vs_baseline": auc_null,
|
||||
"separable_churned_vs_baseline": separable,
|
||||
"null_baseline_vs_baseline": null_ok,
|
||||
"pass": bool(separable and null_ok),
|
||||
}
|
||||
|
||||
|
||||
# --- Item 3: agent-selector reproducibility --------------------------------- #
|
||||
def agent_reproducibility_gate(r: int = 5) -> Dict:
|
||||
"""Same seed + churn history → byte-identical selector decisions for the agent
|
||||
arm (deterministic local heuristic backend). The Ollama confirmatory arm is
|
||||
reproducible via its committed decision-cache replay (tested separately)."""
|
||||
all_identical = True
|
||||
checks: List[Dict] = []
|
||||
for i in range(r):
|
||||
seed = _cal_seed("agent-repro", i)
|
||||
a = _replay(seed, CHURN_KP, strategy="agent")
|
||||
b = _replay(seed, CHURN_KP, strategy="agent")
|
||||
same = (a.initial_circuit == b.initial_circuit
|
||||
and [rb.circuit for rb in a.rebuilds] == [rb.circuit for rb in b.rebuilds])
|
||||
all_identical = all_identical and same
|
||||
checks.append({"seed": seed, "identical": same})
|
||||
return {"runs": r, "checks": checks, "pass": bool(all_identical)}
|
||||
|
||||
|
||||
# --- Item 4: entropy calibration (inherited) -------------------------------- #
|
||||
def entropy_calibration() -> Dict:
|
||||
checks: List[Dict] = []
|
||||
ok = True
|
||||
for n in (2, 4, 8, 16, 50):
|
||||
h = shannon_entropy_bits({f"s{ix}": 1 for ix in range(n)})
|
||||
item_ok = abs(h - math.log2(n)) < 1e-9
|
||||
ok = ok and item_ok
|
||||
checks.append({"N": n, "shannon_bits": h, "log2N": math.log2(n), "equals": item_ok})
|
||||
return {"checks": checks, "pass": bool(ok)}
|
||||
|
||||
|
||||
def calibration_gate(r: int = R_DRY) -> Dict:
|
||||
"""Run all four RQ3 §3-4 gate items on the DRY/offline pass and return a report."""
|
||||
item1 = churn_bites_gate(r)
|
||||
item2 = rebuild_classifier_gate(r)
|
||||
item3 = agent_reproducibility_gate()
|
||||
item4 = entropy_calibration()
|
||||
return {
|
||||
"schema": "sor-rq3-calibration/1",
|
||||
"dry_only": True,
|
||||
"offline_no_engine_no_traffic": True,
|
||||
"no_confirmatory_data_read": True,
|
||||
"R": r,
|
||||
"gate": {
|
||||
"item1_churn_bites": item1,
|
||||
"item2_rebuild_classifier": item2,
|
||||
"item3_agent_reproducibility": item3,
|
||||
"item4_entropy_calibration": item4,
|
||||
"all_pass": bool(item1["pass"] and item2["pass"] and item3["pass"] and item4["pass"]),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
print(json.dumps(calibration_gate(), indent=2, sort_keys=True))
|
||||
sys.exit(0)
|
||||
@@ -126,6 +126,63 @@ def enumerate_rq2p3_cells() -> List[Cell]:
|
||||
return cells
|
||||
|
||||
|
||||
# --- RQ3 companion cells (frozen prereg §3/§4; run-brief params) ------------ #
|
||||
RQ3_KILL_PROB_PCT = 30 # pinned churn kill probability (run-brief §2(B))
|
||||
RQ3_CHURN_STEPS = 20 # pinned churn horizon (run-brief §2(B))
|
||||
|
||||
|
||||
def enumerate_rq3_cells() -> List[Cell]:
|
||||
"""The RQ3 companion cells (`sor-consent-prereg.md` §3/§4, params pinned in
|
||||
`rq3-companion-run-brief.md` §2), kept **separate** from the frozen lead lattice so
|
||||
`enumerate_cells()` stays byte-identical for every lead-pipeline caller.
|
||||
|
||||
The selector arm (`selector ∈ {static, random, agent}`) is the only manipulation,
|
||||
at the 1-house / bridge-off control, under the pinned churn schedule
|
||||
(`kill_prob_pct = 30`, `steps = 20`). ``static`` is the interleaved control baseline;
|
||||
``random`` and ``agent`` are treatments. Tagged ``rq = "RQ3"`` so they are distinct
|
||||
from the frozen RQ1/RQ2 cells and are skipped by the lead collectors.
|
||||
"""
|
||||
churn_tag = f"kp{RQ3_KILL_PROB_PCT}s{RQ3_CHURN_STEPS}"
|
||||
cells: List[Cell] = []
|
||||
for sel in ("static", "random", "agent"):
|
||||
cells.append(Cell(
|
||||
"RQ3",
|
||||
f"RQ3/topo=1house/bridge=off/selector={sel}/churn={churn_tag}",
|
||||
{"bridge": "off", "topology": "1house", "selector": sel,
|
||||
"churn_kill_prob_pct": str(RQ3_KILL_PROB_PCT),
|
||||
"churn_steps": str(RQ3_CHURN_STEPS)},
|
||||
is_control=(sel == "static"),
|
||||
))
|
||||
return cells
|
||||
|
||||
|
||||
def rq3_schedule(order_seed: int, r: int = R_RUNS) -> List[PlannedRun]:
|
||||
"""RQ3 executable run order, honouring §2's interleaving discipline: runs
|
||||
randomized **within** each cell, and the ``static`` control arm interleaved
|
||||
**before and after** the {random, agent} treatments so grid drift is bracketed.
|
||||
Deterministic from ``order_seed`` (a measurement-side ordering seed). Kept
|
||||
**separate** from the frozen lead ``battery_schedule`` (which is left byte-identical)."""
|
||||
rng = random.Random(order_seed)
|
||||
cells = enumerate_rq3_cells()
|
||||
control = next(c for c in cells if c.is_control)
|
||||
treatments = [c for c in cells if not c.is_control]
|
||||
|
||||
def runs_of(c: Cell) -> List[PlannedRun]:
|
||||
rs = [PlannedRun(c.cell_id, c.rq, i, derive_seed(c.cell_id, i), c.is_control)
|
||||
for i in range(r)]
|
||||
rng.shuffle(rs)
|
||||
return rs
|
||||
|
||||
control_runs = runs_of(control)
|
||||
half = len(control_runs) // 2
|
||||
ordered: List[PlannedRun] = list(control_runs[:half]) # control BEFORE
|
||||
treatment_runs = [pr for c in treatments for pr in runs_of(c)]
|
||||
rng.shuffle(treatment_runs)
|
||||
ordered.extend(treatment_runs)
|
||||
ordered.extend(control_runs[half:]) # control AFTER
|
||||
return ordered
|
||||
|
||||
|
||||
def declared_na_cells() -> List[Cell]:
|
||||
"""N/A cells declared at the design (not run, not dropped ad hoc) — padding is
|
||||
only defined for bridge-on, so bridge-off+padding is N/A (§2)."""
|
||||
|
||||
+211
-2
@@ -35,18 +35,21 @@ from __future__ import annotations
|
||||
import hashlib
|
||||
import json
|
||||
import shutil
|
||||
import statistics
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Dict, List, Optional, Sequence, Tuple
|
||||
|
||||
from cmd_chat.sor import battery as sor_battery
|
||||
from cmd_chat.sor.analysis.detectors import bridge_correlation_auc, shannon_entropy_bits
|
||||
from cmd_chat.sor.analysis.metrics import compute_metrics, write_metrics
|
||||
from cmd_chat.sor.analysis.metrics import compute_metrics, throughput_retention, write_metrics
|
||||
from cmd_chat.sor.assembler import assemble
|
||||
from cmd_chat.sor.churn import churn_schedule
|
||||
from cmd_chat.sor.config import Domain, SorRng
|
||||
from cmd_chat.sor.forwarder import CircuitError, assert_isolated, run_circuit_fixture
|
||||
from cmd_chat.sor.provenance import Node, RunManifest, write_manifest
|
||||
from cmd_chat.sor.selector import SelectionResult
|
||||
from cmd_chat.sor.selector import SelectionResult, run_selection
|
||||
|
||||
DEFAULT_BINS = 32
|
||||
|
||||
@@ -331,3 +334,209 @@ def run_battery(
|
||||
path.write_text(json.dumps(doc, indent=2, sort_keys=True) + "\n", encoding="utf-8")
|
||||
doc["_results_path"] = str(path)
|
||||
return doc
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# RQ3 — churn-resilient selector collection.
|
||||
#
|
||||
# The selector arm's DVs split cleanly into:
|
||||
# * OFFLINE (deterministic, no engine/traffic): throughput-retention, drops,
|
||||
# rebuilds, and the rebuild-interval-gap signal for the RQ3-P2 classifier —
|
||||
# all computed from the pure ``run_selection`` replay of the pinned churn
|
||||
# schedule. These need no live circuit and are collected here directly.
|
||||
# * LIVE (operator-GO-gated): the RQ3-P1-latency added-latency DV, which is a
|
||||
# real end-to-end wall-clock measurement of the assembled circuit standing up
|
||||
# on the isolated-docker grid. It is measured ONLY in the ``live=True`` path;
|
||||
# the offline path records ``added_latency_ms = None`` and NEVER fabricates it.
|
||||
# --------------------------------------------------------------------------- #
|
||||
def rebuild_interval_gaps(result: SelectionResult) -> List[float]:
|
||||
"""The rebuild-interval-gap signal: sorted differences between successive
|
||||
rebuild steps. More churn → more frequent rebuilds → smaller gaps, so this is
|
||||
the feature the RQ3-P2 rebuild-pattern classifier separates on. Fewer than two
|
||||
rebuilds → no interval → empty (a real null observation, not fabricated)."""
|
||||
ts = sorted(rb.t for rb in result.rebuilds)
|
||||
return [float(b - a) for a, b in zip(ts, ts[1:])]
|
||||
|
||||
|
||||
def _rq3_pool(cell, size: int = 8) -> List[str]:
|
||||
"""The 1-house consenting-node pool the selector rebuilds over. Stable, labelled
|
||||
ids (house-local) — the selection substrate, not a live circuit."""
|
||||
house = cell.factors.get("topology", "1house")
|
||||
return [f"{house}/node{ix:02d}" for ix in range(size)]
|
||||
|
||||
|
||||
@dataclass
|
||||
class RQ3RunReport:
|
||||
cell_id: str
|
||||
rq: str
|
||||
run_index: int
|
||||
seed: int
|
||||
strategy: str
|
||||
hops: int
|
||||
kill_prob_pct: int
|
||||
steps: int
|
||||
drops: int
|
||||
rebuilds: int
|
||||
deferred: int
|
||||
every_drop_rebuilt: bool
|
||||
throughput_retention: float
|
||||
rebuild_gaps: List[float]
|
||||
added_latency_ms: Optional[float] # measured only in the live path; else None
|
||||
run_dir: str
|
||||
|
||||
|
||||
def run_rq3_cell_run(
|
||||
cell,
|
||||
run_index: int,
|
||||
out_root: Path,
|
||||
*,
|
||||
engine: str = "docker",
|
||||
pool_size: int = 8,
|
||||
hops: int = 3,
|
||||
c_circuits: int = 0,
|
||||
payload_size: int = 4096,
|
||||
live: bool = False,
|
||||
) -> RQ3RunReport:
|
||||
"""Collect one RQ3 (cell, run): replay the pinned churn schedule under the cell's
|
||||
selector strategy and record the OFFLINE selector DVs (retention, drops, rebuilds,
|
||||
rebuild-interval gaps). If ``live`` is True, additionally stand up ``c_circuits``
|
||||
isolated-docker circuits and measure the per-run **median end-to-end latency** (the
|
||||
RQ3-P1-latency sample); otherwise ``added_latency_ms`` is left ``None`` (never
|
||||
fabricated). Writes a write-once ``rq3-run.json`` sidecar. Deterministic offline."""
|
||||
kp = int(cell.factors["churn_kill_prob_pct"])
|
||||
steps = int(cell.factors["churn_steps"])
|
||||
strategy = cell.factors.get("selector", "static")
|
||||
seed = sor_battery.derive_seed(cell.cell_id, run_index)
|
||||
run_dir = Path(out_root) / f"rq3-{cell.cell_id.replace('/', '_')}-r{run_index}"
|
||||
run_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
nodes = _rq3_pool(cell, pool_size)
|
||||
schedule = churn_schedule(seed, nodes, steps, kill_prob_pct=kp)
|
||||
result = run_selection(seed, nodes, hops, schedule, strategy=strategy)
|
||||
gaps = rebuild_interval_gaps(result)
|
||||
|
||||
added_latency_ms: Optional[float] = None
|
||||
if live:
|
||||
# RQ3-P1-latency: a REAL end-to-end measurement, isolated-docker only. Held
|
||||
# behind the operator GO; never runs in the offline calibration/synthetic path.
|
||||
assert_isolated(engine)
|
||||
if shutil.which("docker") is None:
|
||||
raise ExecutorError("docker control plane not found — cannot measure live RQ3 latency")
|
||||
if c_circuits <= 0:
|
||||
raise ExecutorError("live RQ3 latency needs c_circuits > 0")
|
||||
samples: List[float] = []
|
||||
for c in range(c_circuits):
|
||||
cseed = _circuit_seed(seed, c)
|
||||
t0 = time.perf_counter()
|
||||
res = run_circuit_fixture(
|
||||
cseed, engine=engine, hops=hops, out_root=run_dir / "circuits",
|
||||
payload_size=payload_size,
|
||||
)
|
||||
dt_ms = (time.perf_counter() - t0) * 1000.0
|
||||
if not res.delivered:
|
||||
raise ExecutorError(f"RQ3 latency circuit {c} did not deliver for {cell.cell_id}")
|
||||
samples.append(dt_ms)
|
||||
added_latency_ms = statistics.median(samples)
|
||||
|
||||
doc = {
|
||||
"schema": "sor-rq3-run/1",
|
||||
"cell_id": cell.cell_id,
|
||||
"rq": cell.rq,
|
||||
"run_index": run_index,
|
||||
"seed": seed,
|
||||
"selector_strategy": result.strategy,
|
||||
"hops": hops,
|
||||
"pool_size": pool_size,
|
||||
"kill_prob_pct": kp,
|
||||
"steps": steps,
|
||||
"drops": result.drops,
|
||||
"rebuilds": len(result.rebuilds),
|
||||
"deferred": result.deferred,
|
||||
"every_drop_rebuilt": result.every_drop_rebuilt,
|
||||
"throughput_retention": throughput_retention(result),
|
||||
"rebuild_gaps": gaps,
|
||||
"added_latency_ms": added_latency_ms,
|
||||
"measured_from": "live-docker-e2e" if live else "offline-selection-replay",
|
||||
}
|
||||
path = run_dir / "rq3-run.json"
|
||||
if not path.exists():
|
||||
path.write_text(json.dumps(doc, indent=2, sort_keys=True) + "\n", encoding="utf-8")
|
||||
|
||||
return RQ3RunReport(
|
||||
cell_id=cell.cell_id, rq=cell.rq, run_index=run_index, seed=seed,
|
||||
strategy=result.strategy, hops=hops, kill_prob_pct=kp, steps=steps,
|
||||
drops=result.drops, rebuilds=len(result.rebuilds), deferred=result.deferred,
|
||||
every_drop_rebuilt=result.every_drop_rebuilt,
|
||||
throughput_retention=throughput_retention(result), rebuild_gaps=gaps,
|
||||
added_latency_ms=added_latency_ms, run_dir=str(run_dir),
|
||||
)
|
||||
|
||||
|
||||
def run_rq3_battery(
|
||||
out_root: Path,
|
||||
*,
|
||||
engine: str = "docker",
|
||||
order_seed: int = sor_battery.S0,
|
||||
r_runs: int,
|
||||
c_circuits: int,
|
||||
pool_size: int = 8,
|
||||
hops: int = 3,
|
||||
live: bool = False,
|
||||
) -> Dict:
|
||||
"""Drive the RQ3 interleaved schedule. Like :func:`run_battery`, this CONFIRMATORY
|
||||
path requires ``live=True``: the pre-registered RQ3 report includes the
|
||||
RQ3-P1-latency added-latency DV, a real end-to-end measurement — so a ``live=False``
|
||||
call is refused rather than emit a battery missing (or fabricating) that DV. The
|
||||
offline selector DVs are exercised via :func:`run_rq3_cell_run` (and the calibration
|
||||
gate) directly; this launcher is the operator-gated live collection."""
|
||||
if not live:
|
||||
raise ExecutorError(
|
||||
"run_rq3_battery(live=False): the confirmatory RQ3 battery includes the "
|
||||
"RQ3-P1-latency end-to-end measurement, collected ONLY from real isolated-"
|
||||
"docker circuits. Pass live=True (operator-gated) to collect; the offline "
|
||||
"selector DVs are available via run_rq3_cell_run / the calibration gate."
|
||||
)
|
||||
assert_isolated(engine)
|
||||
out_root = Path(out_root)
|
||||
out_root.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
schedule = sor_battery.rq3_schedule(order_seed, r=r_runs)
|
||||
by_cell = {c.cell_id: c for c in sor_battery.enumerate_rq3_cells()}
|
||||
schedule = [pr for pr in schedule if pr.cell_id in by_cell]
|
||||
|
||||
reports: List[RQ3RunReport] = []
|
||||
for pr in schedule:
|
||||
reports.append(run_rq3_cell_run(
|
||||
by_cell[pr.cell_id], pr.run_index, out_root, engine=engine,
|
||||
pool_size=pool_size, hops=hops, c_circuits=c_circuits, live=True,
|
||||
))
|
||||
|
||||
agg: Dict[str, Dict] = {}
|
||||
for rep in reports:
|
||||
a = agg.setdefault(rep.cell_id, {
|
||||
"strategy": rep.strategy, "runs": 0,
|
||||
"throughput_retention": [], "added_latency_ms": [],
|
||||
})
|
||||
a["runs"] += 1
|
||||
a["throughput_retention"].append(rep.throughput_retention)
|
||||
if rep.added_latency_ms is not None:
|
||||
a["added_latency_ms"].append(rep.added_latency_ms)
|
||||
|
||||
doc = {
|
||||
"schema": "sor-rq3-battery-results/1",
|
||||
"engine": engine,
|
||||
"order_seed": order_seed,
|
||||
"r_runs": r_runs,
|
||||
"c_circuits": c_circuits,
|
||||
"pool_size": pool_size,
|
||||
"hops": hops,
|
||||
"measured_from": "live-docker-e2e",
|
||||
"n_runs": len(reports),
|
||||
"cells": agg,
|
||||
"runs": [vars(r) for r in reports],
|
||||
}
|
||||
path = out_root / "rq3-battery-results.json"
|
||||
if not path.exists():
|
||||
path.write_text(json.dumps(doc, indent=2, sort_keys=True) + "\n", encoding="utf-8")
|
||||
doc["_results_path"] = str(path)
|
||||
return doc
|
||||
|
||||
@@ -0,0 +1,139 @@
|
||||
"""RQ3 companion executor-wiring — SYNTHETIC / OFFLINE only
|
||||
(`rq3-companion-run-brief.md` §2-4). No confirmatory battery is run and no
|
||||
confirmatory record is read; these pins exercise the build + the calibration gate.
|
||||
|
||||
Pins: (a) the RQ3 cells + interleaved schedule are well-formed and kept DISJOINT
|
||||
from the frozen lead lattice (which stays byte-identical); (b) the offline selector
|
||||
DVs (throughput-retention, drops/rebuilds, rebuild-interval gaps) are deterministic
|
||||
and never fabricate a live-only added-latency; (c) the confirmatory `run_rq3_battery`
|
||||
refuses to run offline (its added-latency DV is a real measurement); (d) the agent
|
||||
arm replays byte-identically from its committed decision cache (no network); (e) the
|
||||
four §3-4 calibration gate items pass green with real teeth.
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from cmd_chat.sor.agent_selector import OllamaAgentPolicy
|
||||
from cmd_chat.sor.analysis.rq3_calibration import calibration_gate
|
||||
from cmd_chat.sor.battery import (
|
||||
RQ3_CHURN_STEPS,
|
||||
RQ3_KILL_PROB_PCT,
|
||||
derive_seed,
|
||||
enumerate_cells,
|
||||
enumerate_rq3_cells,
|
||||
rq3_schedule,
|
||||
)
|
||||
from cmd_chat.sor.churn import churn_schedule
|
||||
from cmd_chat.sor.executor import (
|
||||
ExecutorError,
|
||||
rebuild_interval_gaps,
|
||||
run_rq3_battery,
|
||||
run_rq3_cell_run,
|
||||
)
|
||||
from cmd_chat.sor.selector import Rebuild, SelectionResult, run_selection
|
||||
|
||||
S0 = 20260719
|
||||
|
||||
|
||||
# --- cells + schedule -------------------------------------------------------- #
|
||||
def test_rq3_cells_shape_and_pinned_churn():
|
||||
cells = enumerate_rq3_cells()
|
||||
assert [c.factors["selector"] for c in cells] == ["static", "random", "agent"]
|
||||
assert all(c.rq == "RQ3" for c in cells)
|
||||
assert [c.is_control for c in cells] == [True, False, False] # static is baseline
|
||||
for c in cells:
|
||||
assert c.factors["topology"] == "1house" and c.factors["bridge"] == "off"
|
||||
assert int(c.factors["churn_kill_prob_pct"]) == RQ3_KILL_PROB_PCT == 30
|
||||
assert int(c.factors["churn_steps"]) == RQ3_CHURN_STEPS == 20
|
||||
|
||||
|
||||
def test_rq3_cells_disjoint_from_frozen_lead_lattice():
|
||||
lead = enumerate_cells()
|
||||
assert len(lead) == 6 # the frozen lead lattice is untouched
|
||||
lead_ids = {c.cell_id for c in lead}
|
||||
assert lead_ids.isdisjoint({c.cell_id for c in enumerate_rq3_cells()})
|
||||
|
||||
|
||||
def test_rq3_schedule_brackets_control_and_is_deterministic():
|
||||
sched = rq3_schedule(S0, r=6)
|
||||
assert len(sched) == 3 * 6
|
||||
control_id = next(c.cell_id for c in enumerate_rq3_cells() if c.is_control)
|
||||
is_ctrl = [pr.cell_id == control_id for pr in sched]
|
||||
# control interleaved BEFORE and AFTER the {random, agent} treatment block
|
||||
assert is_ctrl[0] and is_ctrl[-1]
|
||||
first_treat = is_ctrl.index(False)
|
||||
last_treat = len(is_ctrl) - 1 - is_ctrl[::-1].index(False)
|
||||
assert not any(is_ctrl[first_treat:last_treat + 1]) # treatments contiguous
|
||||
assert rq3_schedule(S0, r=6) == sched # same order_seed -> identical order
|
||||
|
||||
|
||||
# --- rebuild-interval-gap signal -------------------------------------------- #
|
||||
def test_rebuild_interval_gaps_sorted_diffs_and_null():
|
||||
res = SelectionResult("static", 1, 3, ["a", "b", "c"], rebuilds=[
|
||||
Rebuild(5, "x", []), Rebuild(2, "y", []), Rebuild(2, "z", []), Rebuild(9, "w", []),
|
||||
])
|
||||
assert rebuild_interval_gaps(res) == [0.0, 3.0, 4.0] # sorted [2,2,5,9] -> diffs
|
||||
lone = SelectionResult("static", 1, 3, ["a", "b", "c"], rebuilds=[Rebuild(4, "x", [])])
|
||||
assert rebuild_interval_gaps(lone) == [] # <2 rebuilds -> no interval (not fabricated)
|
||||
|
||||
|
||||
# --- offline selector DVs ---------------------------------------------------- #
|
||||
def test_run_rq3_cell_run_offline_dvs_and_no_fabricated_latency(tmp_path):
|
||||
cell = next(c for c in enumerate_rq3_cells() if c.factors["selector"] == "agent")
|
||||
rep = run_rq3_cell_run(cell, 0, tmp_path, live=False)
|
||||
assert rep.added_latency_ms is None # never fabricated in the offline path
|
||||
assert 0.0 <= rep.throughput_retention <= 1.0
|
||||
assert rep.kill_prob_pct == 30 and rep.steps == 20
|
||||
assert rep.drops > 0 and rep.rebuilds > 0 # the pinned churn genuinely bites
|
||||
assert (Path(rep.run_dir) / "rq3-run.json").exists() # write-once DV sidecar
|
||||
|
||||
|
||||
def test_run_rq3_cell_run_is_deterministic(tmp_path):
|
||||
cell = next(c for c in enumerate_rq3_cells() if c.factors["selector"] == "random")
|
||||
a = run_rq3_cell_run(cell, 3, tmp_path / "a", live=False)
|
||||
b = run_rq3_cell_run(cell, 3, tmp_path / "b", live=False)
|
||||
assert (a.seed, a.drops, a.rebuilds, a.deferred) == (b.seed, b.drops, b.rebuilds, b.deferred)
|
||||
assert a.throughput_retention == b.throughput_retention
|
||||
assert a.rebuild_gaps == b.rebuild_gaps
|
||||
|
||||
|
||||
def test_run_rq3_battery_refuses_offline(tmp_path):
|
||||
with pytest.raises(ExecutorError):
|
||||
run_rq3_battery(tmp_path, r_runs=2, c_circuits=1, live=False)
|
||||
|
||||
|
||||
# --- agent arm: committed-cache replay is byte-identical (no network) -------- #
|
||||
def test_agent_cache_replay_is_reproducible():
|
||||
nodes = [f"1house/node{i:02d}" for i in range(8)]
|
||||
seed = derive_seed("rq3-cache-replay", 0)
|
||||
sched = churn_schedule(seed, nodes, 20, kill_prob_pct=30)
|
||||
# host is deliberately unreachable → model query fails → deterministic heuristic
|
||||
# fallback; the decisions are cached as they are made.
|
||||
pol1 = OllamaAgentPolicy(host="http://127.0.0.1:9", timeout=0.05)
|
||||
first = run_selection(seed, nodes, 3, sched, policy=pol1)
|
||||
cache = dict(pol1._cache)
|
||||
assert cache # decisions were recorded
|
||||
|
||||
# replay from the committed cache alone → identical circuits, all cache-sourced.
|
||||
pol2 = OllamaAgentPolicy(host="http://127.0.0.1:9", timeout=0.05, cache=cache)
|
||||
second = run_selection(seed, nodes, 3, sched, policy=pol2)
|
||||
assert second.initial_circuit == first.initial_circuit
|
||||
assert [rb.circuit for rb in second.rebuilds] == [rb.circuit for rb in first.rebuilds]
|
||||
assert all(d["source"] == "cache" for d in pol2.decisions())
|
||||
|
||||
|
||||
# --- calibration gate has real teeth (§3-4) ---------------------------------- #
|
||||
def test_rq3_calibration_gate_all_items_pass_with_teeth():
|
||||
report = calibration_gate()
|
||||
gate = report["gate"]
|
||||
assert gate["all_pass"] is True
|
||||
bites = gate["item1_churn_bites"]
|
||||
assert bites["total_drops"] > 0 and bites["total_rebuilds"] > 0
|
||||
clf = gate["item2_rebuild_classifier"]
|
||||
assert clf["separable_churned_vs_baseline"] and clf["null_baseline_vs_baseline"]
|
||||
assert clf["auc_churned_vs_baseline"] >= 0.90 # regime signal has teeth
|
||||
assert abs(clf["auc_baseline_vs_baseline"] - 0.5) <= 0.15 # same-regime null is blind
|
||||
assert gate["item3_agent_reproducibility"]["pass"]
|
||||
assert gate["item4_entropy_calibration"]["pass"]
|
||||
Reference in New Issue
Block a user