diff --git a/OVERSEER-STATUS.md b/OVERSEER-STATUS.md index 446cb5f..d9de989 100644 --- a/OVERSEER-STATUS.md +++ b/OVERSEER-STATUS.md @@ -106,3 +106,6 @@ RQ2 RATIFIED (operator Andre, 2026-07-20, while blind) — observation-consisten - RUNNING (do NOT trim): `executor.run_rq3_battery(live=True)` collecting the full frozen schedule — selector∈{static,random,agent} × pinned churn kp30/steps20, R=30 × C=50 = 4,500 real isolated-docker circuits (~12h est. at ~9.8s/circuit). Launch dir `output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/` (gitignored; force-add anchors on completion). Verified genuinely progressing: python PID 51087 alive + live 4-container circuit (client+3 hops) up on docker. **Completion-detection: `rq3-battery-results.json` written once at end; progress = count of `rq3-*/rq3-run.json` sidecars → 90 (3 cells × 30 runs).** Containment intact (isolated-docker only, self-traffic, $0 — local heuristic/random arms; no frontier spend). Both prereg SHAs intact; worktree-only. Analysis (RQ3-P1-perf/latency, RQ3-P2, Holm-7) runs AFTER completion on the sealed record. - 2026-07-21 COMPANION PAPER — RQ2-P3 HALF FILLED POST-SEAL (D3 combined paper). `docs/stage-07-companion-methods.md`: §3 un-marked FREEZE-PENDING → FROZEN 2026-07-21 (SHA 8db4e8a7…); §5 Results + §6 Discussion for RQ2-P3 filled FROM THE SEALED RECORD ONLY (H1 ρ=+0.6244 CI[+0.5941,+0.6545]; H2 β=+0.7052 CI[+0.6195,+0.7903] n=270; H3 RESOLVED=MIX; Holm own {H1,H2} both reject — effect+CI, never bare p). Finding stated plainly: shared-pool concentration RAISES anonymity (mix), REFUTES naive funnel; framed as honest QUALIFICATION/CORRECTION of lead RQ2-P1 "shrink" as a unique-bridge (fresh-bridge-per-circuit) artifact — cites `docs/note-unique-bridge-artifact.md` + frozen mechanism prereg SHA 8db4e8a7…. MANDATORY disclosure carried: §7 dry-pass previewed direction (ρ 0→+0.838); confirmatory quantifies effect already visible at calibration; pre-committed hypotheses were two-sided. - BLIND HOLD (RQ3): RQ3 Results/Discussion AND authoritative Holm-7 LEFT AS HELD-BLIND placeholders — RQ3 battery still running (progress read by sidecar COUNT only = 3/90; NO rq3-run.json contents read). Lead RQ2-P1 NOT re-litigated (companion QUALIFIES; committed lead paper + frozen prereg f22331a72e… untouched). This stone: $0/offline, worktree-only. NEXT: HOLD for RQ3 — no new stones, no peeking at RQ3 data until `rq3-battery-results.json` lands. +- 2026-07-22 RQ3 BATTERY COMPLETE (90/90 runs, live-docker-e2e, exited clean) → UN-BLIND + FROZEN analysis. RAW SEALED: `.../20260722T040640Z/confirmatory-data/SHA256SUMS` (battery-results.json `5b61e461…` + 90 rq3-run.json sidecars). New analyzer `analysis/rq3_confirm.py` (RQ3 analogue of rq2p3_confirm; frozen `stats`/`metrics` UNTOUCHED — run-level multi-arm bootstrap mirrors `two_sample_diff_ci`, 10k BCa, α=0.05). Sealed analysis `.../20260722T040640Z/analysis/rq3-confirmatory-analysis.json` (SHA `e09c66ef…`). Tests `tests/test_sor_rq3_confirm.py` 6 passed; full SOR suite 207 passed (no regression). +- RQ3 RESULT (effect+CI, never bare p): **RQ3-P1-perf FAIL/H0** — retention margin agent−max(static,random) = **−0.6pp**, BCa CI [−1.58pp, +0.39pp]; every selector heals ~all churn drops (~99% retention) so no ≥+10pp agent gain. **RQ3-P1-latency HOLDS budget** — added-latency(agent−min-baseline=random) = **−13.5ms**, CI [−52.1, +34.9]ms, upper ≤100ms (agent not slower). **RQ3-P2 FAIL/not-excluded** — rebuild-classifier AUC(agent vs pooled baseline, per-run mean-gap) = **0.587**, CI [0.458, 0.703], upper 0.703 > 0.60 → fingerprint NOT excluded (underpowered at n=30). **RQ3-P3 = H0** (P1-perf fails ∧ P2 fails). Honest null: the local agent selector neither beats baselines nor is certifiably non-classifiable. +- AUTHORITATIVE HOLM-7 (frozen family size=7; supersedes lead's conservative partial embedding, D3). RQ2-P3 slot carries the mechanism-corrected primary **H1-pooled Spearman p=0** (not the lead's degenerate p=1). **SURVIVORS: RQ1-P1 (r1×7), RQ2-P1 shrink (r2×6), RQ2-P3 mix (r3×5)** all holm_p=0. NON-survivors: RQ1-P2 (r4×4, holm_p=0.365), RQ3-P2 (r5×3, 0.511), RQ3-P1-perf (r6×2, 0.511), RQ3-P1-latency (r7×1, 0.511). Lead RQ1-P1 & RQ2-P1 survive regardless. Both prereg SHAs intact; $0/offline; worktree-only. NEXT: fill RQ3 half of companion paper (stone 2). diff --git a/cmd_chat/sor/analysis/rq3_confirm.py b/cmd_chat/sor/analysis/rq3_confirm.py new file mode 100644 index 0000000..7d6eac4 --- /dev/null +++ b/cmd_chat/sor/analysis/rq3_confirm.py @@ -0,0 +1,310 @@ +"""RQ3 confirmatory analysis — reads the SEALED live-docker battery, applies the +FROZEN prereg §6 plan, and computes the authoritative Holm-7 over the whole family. + +This is the RQ3 analogue of ``rq2p3_confirm.py``: it does **not** re-specify anything. +It loads the immutable ``rq3-battery-results.json`` (the operator-gated live-docker +run) and the pinned params in ``docs/rq3-companion-run-brief.md`` and computes the three +frozen RQ3 confirmatory tests, each as an **effect + BCa 95% CI** (never a bare p; the p +is carried only to order the Holm family, per prereg §6): + + * **RQ3-P1-perf** — throughput-retention margin: mean retention(agent) − + max(mean retention(static), mean retention(random)); perf holds iff the CI **lower** + bound ≥ +10 pp (frozen gate). + * **RQ3-P1-latency** — added-latency(agent) = median e2e latency(agent) − + min(median latency(static), median latency(random)) [the faster / min-latency baseline + arm, run-brief §3.2]; anonymity-latency budget holds iff the CI **upper** bound ≤ 100 ms. + * **RQ3-P2** — rebuild-classifier AUC separating the agent selector's per-run mean + inter-rebuild-gap signal from the pooled baseline selectors' signal (the fingerprint + question, §1(b)); anonymity holds iff the CI **upper** bound ≤ 0.60. + * **RQ3-P3** — logical AND: CONFIRM iff (P1-perf ∧ P1-latency) ∧ P2; else H0. + +The BCa machinery is the frozen ``stats`` toolkit — this harness only *drives* it at the +run level (resample whole runs, the confirmatory grouping unit) via a generic multi-arm +bootstrap that mirrors ``stats.two_sample_diff_ci`` (independent per-arm resampling, a +combined leave-one-out jackknife, ``stats._bca_endpoints``). No frozen instrument is edited. + +**Authoritative Holm-7.** Once all seven confirmatory p-values exist, this computes the +exact ``stats.holm_bonferroni`` step-down over the frozen size-7 family +{RQ1-P1, RQ1-P2, RQ2-P1, RQ2-P3, RQ3-P1-perf, RQ3-P1-latency, RQ3-P2}. The four prior +p-values are read from the SEALED lead record (RQ1-P1, RQ1-P2, RQ2-P1) and the SEALED +RQ2-P3′ mechanism record (RQ2-P3 slot = its primary H1-pooled Spearman ρ — the direct +operationalization of the frozen single-slot "Spearman ρ between concentration and H"; +the lead's degenerate as-instrumented RQ2-P3 is superseded by the mechanism-corrected +result, per operator decision D3). This Holm-7 is the authoritative final correction and +supersedes the lead paper's deliberately conservative *partial* embedding; both remain +valid, the partial never under-corrects. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import statistics +import sys +from collections import defaultdict +from pathlib import Path +from typing import Callable, Dict, List, Sequence + +from cmd_chat.sor.analysis import stats +from cmd_chat.sor.analysis.metrics import rebuild_classifier_auc + +# --- frozen family + gates -------------------------------------------------- # +FAMILY_SIZE = 7 +FROZEN_FAMILY = ( + "RQ1-P1", "RQ1-P2", "RQ2-P1", "RQ2-P3", + "RQ3-P1-perf", "RQ3-P1-latency", "RQ3-P2", +) +PERF_MARGIN_MIN_PP = 0.10 # RQ3-P1-perf gate: CI lower ≥ +10 pp +LATENCY_MAX_MS = 100.0 # RQ3-P1-latency gate: CI upper ≤ 100 ms +AUC_CEILING = 0.60 # RQ3-P2 gate: CI upper ≤ 0.60 + +# sealed prior records (immutable) that carry the 4 non-RQ3 family p-values +LEAD_RESULTS = "output/sor-confirmatory/20260720T060132Z/analysis/stage06-results.json" +RQ2P3_RESULTS = "output/sor-rq2p3-confirmatory/rq2p3-confirmatory-results.json" + + +def _seed(tag: str) -> int: + """Deterministic, auditable per-test resampling seed.""" + return int.from_bytes(hashlib.sha256(f"sor-rq3-confirm|{tag}".encode()).digest()[:8], "big") + + +# --- generic run-level multi-arm bootstrap (mirrors stats.two_sample_diff_ci) --- # +def _multi_arm_bootstrap( + arms: Dict[str, Sequence[float]], + statistic: Callable[[Dict[str, List[float]]], float], + *, + null: float, + seed: int, + n_resamples: int = stats.DEFAULT_RESAMPLES, + alpha: float = stats.DEFAULT_ALPHA, +): + """Bootstrap CI + Holm-ordering p for a statistic over >=1 independent arms. + + Each arm is resampled with replacement independently (the frozen two-sample rule + generalized to N arms); BCa uses a combined leave-one-out jackknife (each point + dropped from its OWN arm, others full), exactly as ``stats.two_sample_diff_ci``. + The p is ``stats.two_sided_bootstrap_p`` against ``null`` — carried only to order + the Holm family, never as a stand-alone decision. + """ + import random + + names = list(arms) + data = {k: list(v) for k, v in arms.items()} + theta_hat = float(statistic(data)) + rng = random.Random(seed) + thetas: List[float] = [] + for _ in range(n_resamples): + sample = {k: [v[rng.randrange(len(v))] for _ in range(len(v))] for k, v in data.items()} + thetas.append(float(statistic(sample))) + thetas.sort() + + # combined jackknife: drop point i from arm `name`, keep the others full + can_bca = all(len(v) > 1 for v in data.values()) + if can_bca: + jack: List[float] = [] + for name in names: + n = len(data[name]) + for i in range(n): + d = dict(data) + d[name] = [data[name][j] for j in range(n) if j != i] + jack.append(float(statistic(d))) + q_lo, q_hi, used = stats._bca_endpoints(thetas, theta_hat, jack, alpha) + else: + q_lo, q_hi, used = alpha / 2.0, 1.0 - alpha / 2.0, "percentile" + + ci = stats.CIResult(theta_hat, stats._percentile(thetas, q_lo), + stats._percentile(thetas, q_hi), alpha, n_resamples, used, seed) + p = stats.two_sided_bootstrap_p(thetas, null) + return ci, p + + +# --- battery loading -------------------------------------------------------- # +def load_battery(path: Path) -> Dict[str, Dict[str, List[float]]]: + """Load the sealed battery into per-arm {retention, latency, gap_signal}. + + ``gap_signal`` is the per-run MEAN inter-rebuild gap — the confirmatory grouping + unit (same unit the frozen calibration gate used); runs with no gap are omitted + (no fabricated interval). Retention/latency are the per-run DVs as measured. + """ + doc = json.loads(Path(path).read_text()) + arms: Dict[str, Dict[str, List[float]]] = {} + for _cid, cell in doc["cells"].items(): + arms[cell["strategy"]] = { + "retention": list(cell["throughput_retention"]), + "latency": list(cell["added_latency_ms"]), + "gap_signal": [], + } + gap: Dict[str, List[float]] = defaultdict(list) + for run in doc["runs"]: + strat = run["cell_id"].split("selector=")[1].split("/")[0] + gaps = run.get("rebuild_gaps") or [] + if gaps: + gap[strat].append(statistics.fmean(gaps)) + for strat, sig in gap.items(): + arms[strat]["gap_signal"] = sig + return arms + + +# --- the three frozen RQ3 tests --------------------------------------------- # +def _perf_margin(a: Dict[str, List[float]]) -> float: + return stats.mean(a["agent"]) - max(stats.mean(a["static"]), stats.mean(a["random"])) + + +def _added_latency(a: Dict[str, List[float]]) -> float: + # added-latency over the MIN-latency (faster) baseline arm, run-brief §3.2 + return statistics.median(a["agent"]) - min(statistics.median(a["static"]), statistics.median(a["random"])) + + +def _p2_auc(a: Dict[str, List[float]]) -> float: + # fingerprint question: agent's per-run rebuild-gap signal vs the pooled baseline + return rebuild_classifier_auc(a["agent"], a["baseline"]) + + +def analyze_rq3(arms: Dict[str, Dict[str, List[float]]], *, n_resamples: int = stats.DEFAULT_RESAMPLES) -> Dict: + ret = {k: arms[k]["retention"] for k in ("agent", "static", "random")} + lat = {k: arms[k]["latency"] for k in ("agent", "static", "random")} + pool_baseline = arms["static"]["gap_signal"] + arms["random"]["gap_signal"] + p2_arms = {"agent": arms["agent"]["gap_signal"], "baseline": pool_baseline} + + perf_ci, perf_p = _multi_arm_bootstrap(ret, _perf_margin, null=0.0, + seed=_seed("perf"), n_resamples=n_resamples) + lat_ci, lat_p = _multi_arm_bootstrap(lat, _added_latency, null=0.0, + seed=_seed("latency"), n_resamples=n_resamples) + p2_ci, p2_p = _multi_arm_bootstrap(p2_arms, _p2_auc, null=0.5, + seed=_seed("p2-auc"), n_resamples=n_resamples) + + perf_hold = perf_ci.lo >= PERF_MARGIN_MIN_PP # gate: CI lower ≥ +10 pp + lat_hold = lat_ci.hi <= LATENCY_MAX_MS # gate: CI upper ≤ 100 ms + p2_hold = p2_ci.hi <= AUC_CEILING # gate: CI upper ≤ 0.60 + p1_hold = perf_hold and lat_hold + p3_confirm = p1_hold and p2_hold + + which_lat_baseline = "random" if statistics.median(lat["random"]) <= statistics.median(lat["static"]) else "static" + return { + "RQ3-P1-perf": { + "effect": "throughput_retention_margin_agent_minus_max_baseline", + **perf_ci.as_dict(), "p_for_holm": perf_p, + "gate": f"CI lower ≥ +{PERF_MARGIN_MIN_PP}", "holds": bool(perf_hold), + "decision": "perf-gain" if perf_hold else "no-perf-gain", + }, + "RQ3-P1-latency": { + "effect": "added_latency_ms_agent_minus_min_baseline", + "min_latency_baseline_arm": which_lat_baseline, + **lat_ci.as_dict(), "p_for_holm": lat_p, + "gate": f"CI upper ≤ {LATENCY_MAX_MS} ms", "holds": bool(lat_hold), + "decision": "within-latency-budget" if lat_hold else "over-latency-budget", + }, + "RQ3-P2": { + "effect": "rebuild_classifier_auc_agent_vs_pooled_baseline", + "grouping_unit": "per-run mean inter-rebuild gap", + "n_agent_runs": len(p2_arms["agent"]), "n_baseline_runs": len(p2_arms["baseline"]), + **p2_ci.as_dict(), "p_for_holm": p2_p, + "gate": f"CI upper ≤ {AUC_CEILING}", "holds": bool(p2_hold), + "decision": "no-usable-fingerprint" if p2_hold else "fingerprint-not-excluded", + }, + "RQ3-P3-joint": { + "rule": "CONFIRM iff (P1-perf ∧ P1-latency) ∧ P2", + "p1_holds": bool(p1_hold), "p2_holds": bool(p2_hold), + "confirm": bool(p3_confirm), + "decision": "agent-helps-without-fingerprint" if p3_confirm else "H0", + }, + } + + +# --- authoritative Holm-7 --------------------------------------------------- # +def _load_prior_pvalues(lead_path: Path, rq2p3_path: Path) -> Dict[str, Dict]: + lead = json.loads(Path(lead_path).read_text())["confirmatory"] + rq2p3 = json.loads(Path(rq2p3_path).read_text())["results"] + h1 = rq2p3["H1_pooled_spearman"] + return { + "RQ1-P1": {"p": lead["RQ1-P1"]["p_for_holm"], "source": "sealed lead RQ1-P1"}, + "RQ1-P2": {"p": lead["RQ1-P2"]["p_for_holm"], "source": "sealed lead RQ1-P2"}, + "RQ2-P1": {"p": lead["RQ2-P1"]["p_for_holm"], "source": "sealed lead RQ2-P1 (shrink)"}, + "RQ2-P3": {"p": h1["p_for_holm"], + "source": "sealed RQ2-P3′ H1-pooled Spearman ρ (mechanism-corrected, mix) " + "— supersedes the lead's degenerate as-instrumented RQ2-P3"}, + } + + +def authoritative_holm7(rq3: Dict, lead_path: Path, rq2p3_path: Path) -> Dict: + prior = _load_prior_pvalues(lead_path, rq2p3_path) + pvals = { + "RQ1-P1": prior["RQ1-P1"]["p"], + "RQ1-P2": prior["RQ1-P2"]["p"], + "RQ2-P1": prior["RQ2-P1"]["p"], + "RQ2-P3": prior["RQ2-P3"]["p"], + "RQ3-P1-perf": rq3["RQ3-P1-perf"]["p_for_holm"], + "RQ3-P1-latency": rq3["RQ3-P1-latency"]["p_for_holm"], + "RQ3-P2": rq3["RQ3-P2"]["p_for_holm"], + } + assert set(pvals) == set(FROZEN_FAMILY), "family must be exactly the frozen size-7" + holm = stats.holm_bonferroni(pvals, family_size=FAMILY_SIZE) + rows = [{"name": h.name, "raw_p": h.p, "holm_p": h.p_adjusted, + "multiplier": h.multiplier, "rank": h.rank, "reject": h.reject} for h in holm] + return { + "family_size": FAMILY_SIZE, + "family": list(FROZEN_FAMILY), + "p_sources": {k: prior[k]["source"] for k in prior}, + "rows": rows, + "survivors": [r["name"] for r in rows if r["reject"]], + "non_survivors": [r["name"] for r in rows if not r["reject"]], + "supersedes": "lead paper's conservative partial embedding (report-4 of family-of-7); " + "both valid, partial never under-corrects; RQ1-P1 and RQ2-P1 survive regardless", + } + + +def run(*, battery_path: Path, lead_path: Path, rq2p3_path: Path, + n_resamples: int = stats.DEFAULT_RESAMPLES) -> Dict: + arms = load_battery(battery_path) + rq3 = analyze_rq3(arms, n_resamples=n_resamples) + holm7 = authoritative_holm7(rq3, lead_path, rq2p3_path) + battery_sha = hashlib.sha256(Path(battery_path).read_bytes()).hexdigest() + return { + "schema": "sor-rq3-confirmatory/1", + "measured_from": "live-docker-e2e", + "battery_results_path": str(battery_path), + "battery_results_sha256": battery_sha, + "n_runs_per_arm": {k: len(arms[k]["retention"]) for k in arms}, + "frozen_lead_prereg_sha256": "f22331a72e0d0ccf38b787e63acabbe9d666456ec76076787a6d545c3193425b", + "results": rq3, + "authoritative_holm7": holm7, + "honest_disclosure": ( + "Every RQ3 test reports effect + BCa 95% CI; p is carried ONLY to order the Holm " + "family (prereg §6). Nulls are results: a selector that does not beat baselines, or " + "a rebuild pattern that is not certifiably non-classifiable, is the finding — not spun. " + "Reproducibility caveat (accepted, run-brief §2A): the agent (qwen2.5:3b via local " + "Ollama, temp 0) is reproducible via its committed decision-log + (seed, state-hash) " + "cache replay, NOT via independent model re-execution on other hardware." + ), + } + + +def _seal(out_dir: Path, report: Dict) -> Path: + out_dir.mkdir(parents=True, exist_ok=True) + path = out_dir / "rq3-confirmatory-analysis.json" + path.write_text(json.dumps(report, indent=2, sort_keys=True) + "\n", encoding="utf-8") + digest = hashlib.sha256(path.read_bytes()).hexdigest() + (out_dir / "SHA256SUMS").write_text(f"{digest} {path.name}\n", encoding="utf-8") + return path + + +def main(argv=None) -> int: + ap = argparse.ArgumentParser(prog="python -m cmd_chat.sor.analysis.rq3_confirm") + ap.add_argument("--battery", required=True) + ap.add_argument("--lead", default=LEAD_RESULTS) + ap.add_argument("--rq2p3", default=RQ2P3_RESULTS) + ap.add_argument("--out", default=None) + ap.add_argument("--n-resamples", type=int, default=stats.DEFAULT_RESAMPLES) + args = ap.parse_args(argv) + report = run(battery_path=Path(args.battery), lead_path=Path(args.lead), + rq2p3_path=Path(args.rq2p3), n_resamples=args.n_resamples) + if args.out: + path = _seal(Path(args.out), report) + print(f"sealed -> {path}", file=sys.stderr) + print(json.dumps(report, indent=2, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/output/sor-rq3-confirmatory/20260722T040640Z/analysis/SHA256SUMS b/output/sor-rq3-confirmatory/20260722T040640Z/analysis/SHA256SUMS new file mode 100644 index 0000000..aa4c098 --- /dev/null +++ b/output/sor-rq3-confirmatory/20260722T040640Z/analysis/SHA256SUMS @@ -0,0 +1 @@ +e09c66efe00524ae3e5373bd20121e6e022e6885596f4fa9aabd51653c76929b rq3-confirmatory-analysis.json diff --git a/output/sor-rq3-confirmatory/20260722T040640Z/analysis/rq3-confirmatory-analysis.json b/output/sor-rq3-confirmatory/20260722T040640Z/analysis/rq3-confirmatory-analysis.json new file mode 100644 index 0000000..02949d5 --- /dev/null +++ b/output/sor-rq3-confirmatory/20260722T040640Z/analysis/rq3-confirmatory-analysis.json @@ -0,0 +1,156 @@ +{ + "authoritative_holm7": { + "family": [ + "RQ1-P1", + "RQ1-P2", + "RQ2-P1", + "RQ2-P3", + "RQ3-P1-perf", + "RQ3-P1-latency", + "RQ3-P2" + ], + "family_size": 7, + "non_survivors": [ + "RQ1-P2", + "RQ3-P2", + "RQ3-P1-perf", + "RQ3-P1-latency" + ], + "p_sources": { + "RQ1-P1": "sealed lead RQ1-P1", + "RQ1-P2": "sealed lead RQ1-P2", + "RQ2-P1": "sealed lead RQ2-P1 (shrink)", + "RQ2-P3": "sealed RQ2-P3\u2032 H1-pooled Spearman \u03c1 (mechanism-corrected, mix) \u2014 supersedes the lead's degenerate as-instrumented RQ2-P3" + }, + "rows": [ + { + "holm_p": 0.0, + "multiplier": 7, + "name": "RQ1-P1", + "rank": 1, + "raw_p": 0.0, + "reject": true + }, + { + "holm_p": 0.0, + "multiplier": 6, + "name": "RQ2-P1", + "rank": 2, + "raw_p": 0.0, + "reject": true + }, + { + "holm_p": 0.0, + "multiplier": 5, + "name": "RQ2-P3", + "rank": 3, + "raw_p": 0.0, + "reject": true + }, + { + "holm_p": 0.3648, + "multiplier": 4, + "name": "RQ1-P2", + "rank": 4, + "raw_p": 0.0912, + "reject": false + }, + { + "holm_p": 0.5112, + "multiplier": 3, + "name": "RQ3-P2", + "rank": 5, + "raw_p": 0.1704, + "reject": false + }, + { + "holm_p": 0.5112, + "multiplier": 2, + "name": "RQ3-P1-perf", + "rank": 6, + "raw_p": 0.177, + "reject": false + }, + { + "holm_p": 0.5112, + "multiplier": 1, + "name": "RQ3-P1-latency", + "rank": 7, + "raw_p": 0.497, + "reject": false + } + ], + "supersedes": "lead paper's conservative partial embedding (report-4 of family-of-7); both valid, partial never under-corrects; RQ1-P1 and RQ2-P1 survive regardless", + "survivors": [ + "RQ1-P1", + "RQ2-P1", + "RQ2-P3" + ] + }, + "battery_results_path": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-battery-results.json", + "battery_results_sha256": "5b61e461004722985c1a7a9bc5fdfe855395df3180c71e6efdc3531b8ecf8039", + "frozen_lead_prereg_sha256": "f22331a72e0d0ccf38b787e63acabbe9d666456ec76076787a6d545c3193425b", + "honest_disclosure": "Every RQ3 test reports effect + BCa 95% CI; p is carried ONLY to order the Holm family (prereg \u00a76). Nulls are results: a selector that does not beat baselines, or a rebuild pattern that is not certifiably non-classifiable, is the finding \u2014 not spun. Reproducibility caveat (accepted, run-brief \u00a72A): the agent (qwen2.5:3b via local Ollama, temp 0) is reproducible via its committed decision-log + (seed, state-hash) cache replay, NOT via independent model re-execution on other hardware.", + "measured_from": "live-docker-e2e", + "n_runs_per_arm": { + "agent": 30, + "random": 30, + "static": 30 + }, + "results": { + "RQ3-P1-latency": { + "alpha": 0.05, + "ci_hi": 34.911185735836625, + "ci_lo": -52.14837333187461, + "decision": "within-latency-budget", + "effect": "added_latency_ms_agent_minus_min_baseline", + "gate": "CI upper \u2264 100.0 ms", + "holds": true, + "method": "bca", + "min_latency_baseline_arm": "random", + "n_resamples": 10000, + "p_for_holm": 0.497, + "point": -13.491572928614914, + "seed": 5867095940896968561 + }, + "RQ3-P1-perf": { + "alpha": 0.05, + "ci_hi": 0.003865947497819883, + "ci_lo": -0.015830325279388435, + "decision": "no-perf-gain", + "effect": "throughput_retention_margin_agent_minus_max_baseline", + "gate": "CI lower \u2265 +0.1", + "holds": false, + "method": "bca", + "n_resamples": 10000, + "p_for_holm": 0.177, + "point": -0.006312034914976117, + "seed": 16001687329924348899 + }, + "RQ3-P2": { + "alpha": 0.05, + "ci_hi": 0.7034362881655691, + "ci_lo": 0.45805555555555555, + "decision": "fingerprint-not-excluded", + "effect": "rebuild_classifier_auc_agent_vs_pooled_baseline", + "gate": "CI upper \u2264 0.6", + "grouping_unit": "per-run mean inter-rebuild gap", + "holds": false, + "method": "bca", + "n_agent_runs": 30, + "n_baseline_runs": 60, + "n_resamples": 10000, + "p_for_holm": 0.1704, + "point": 0.5869444444444445, + "seed": 9396385129919407505 + }, + "RQ3-P3-joint": { + "confirm": false, + "decision": "H0", + "p1_holds": false, + "p2_holds": false, + "rule": "CONFIRM iff (P1-perf \u2227 P1-latency) \u2227 P2" + } + }, + "schema": "sor-rq3-confirmatory/1" +} diff --git a/output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/SHA256SUMS b/output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/SHA256SUMS new file mode 100644 index 0000000..fa21afd --- /dev/null +++ b/output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/SHA256SUMS @@ -0,0 +1,91 @@ +5b61e461004722985c1a7a9bc5fdfe855395df3180c71e6efdc3531b8ecf8039 rq3-battery-results.json +110bc93cd92f43f679f108d34e16ca97af1a45076167597509caf431f539eb0d rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r0/rq3-run.json +80e238396a0c4359da81ac4a32d8da645c7bc045908d3c9e0ab6fbb47fd31c05 rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r1/rq3-run.json +c1c41673ab235dde0c0e0ce968c0ca8b6876e37425486e5789ea8da2a9e2ae56 rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r10/rq3-run.json +3e3609b9e78cb26bb5202e9f8f77acd4469a35ea2fc925a25c8f3c449c0598ee rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r11/rq3-run.json +add02b8c1fe94a27e7cfcfdd2017709a572fc635859ea7e9fa6a9aeb7752b179 rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r12/rq3-run.json +a5fc0270f9e87a25ffbf3fbf41dc741fcb1aa981b7312249821c93d666786d19 rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r13/rq3-run.json +72600ffe7378da5b04c8b7cc16c3618d11b1cce6ab3e166a5d58a76379bd3602 rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r14/rq3-run.json +259543832d3d394379849b1c2736962119b290f8b3cad7aa05562bf5f18390cd rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r15/rq3-run.json +d78e5306cae58b7bc46f80244752090cac60102bd037d68d76fe2740b53ebd1e rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r16/rq3-run.json +9bffbb317df459bb7847dfc58c5462066ace4a4befad3ff5919f0986621673eb rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r17/rq3-run.json +d91c00e81cec24fa922a0fdcedc5c5745293d4522b4b2a5b4f3c7df6fe17a179 rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r18/rq3-run.json +f293fe699c1aab60c9650eac031aedc4a605bb62c00cdb92794e2b768e2e8a50 rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r19/rq3-run.json +ec661f5c52133cf814428bd954cea3d1f6118c35c36cb01640573530e1c35f26 rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r2/rq3-run.json +ee1fa456439bdc1c8a8264252b11446a212fda555ac26688c885dbcf5d0d4b65 rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r20/rq3-run.json +bb710e7fd11a50bc262665e857d591645520b47dc4ba9afbd15b770c7b36fa85 rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r21/rq3-run.json +7c92a5ec490f8c40abd7d4d5d05effb00d5c106567426016a17a52cbbde32a28 rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r22/rq3-run.json +cc57bded029cd9f7bb9bc61beef7a2eaecdad798a9872fa570986e10291b92a7 rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r23/rq3-run.json +6d8318fa57e16597d500e6b29661843db532f09cb303e409051611befb8d03e5 rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r24/rq3-run.json +95e2a0df7e320ad3aeec056eb8c6fe872789311b3fc2d85f3b0730fc1984c6e1 rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r25/rq3-run.json +3a036309872aa31fcf150074097226da5bac6485f10786e25313825b983a3fc6 rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r26/rq3-run.json +ad59b9aad30cd10be94c1bf0bd58716a3272fd2e75e1da58aaed5ecaca64bfd0 rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r27/rq3-run.json +0b0c00ddc3db49aa0c85a1b3dc29292435cb53c3753446edd351205115d38f54 rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r28/rq3-run.json +99e28987eaa6da592df7eddcc1134a0ee09bd833f79af54ebe3b6f2609cf256f rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r29/rq3-run.json +86cb773d2315ac118790289779337514e80006817c0c1ec07e0f8d99adaabf6c rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r3/rq3-run.json +08ad295c823da04a6ccbdd758d638c4aae4c7220e9cf4361c81672d800e4bef8 rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r4/rq3-run.json +c7daf7fd45aad4c66570875a41aed8d7c69d0d1246f3291d7983b894d547a6e1 rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r5/rq3-run.json +ae47e7fce7b8acbd36567e31ae1fa5879d5fa483c6d293e0ae588ebf46d925e7 rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r6/rq3-run.json +ce234e3ed894d1c7ba2f6b650c50aec42a3bd423e894f422a64ff1b3a57b3376 rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r7/rq3-run.json +723d66a199e09c13de8735b876db87145e5a3950aae9d749ef6abb91fa6f2af9 rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r8/rq3-run.json +f30da953eb66d498d4e2be2607e8a1d47a4ac3f626f1cc891852c8ef07672919 rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r9/rq3-run.json +f276456e1cfc914fcdb963f5d85e72a33d2858369547d4d11e8da739a2c315e8 rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r0/rq3-run.json +ebd567bb9b129d75d7e794b4f06fa1340263fe7c4abf2cc68546ab476e8e91b9 rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r1/rq3-run.json +761a36d6ec339b84d465ffc9dcd88103e6b0130fb9b1ba4aebeb534a8bf581a4 rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r10/rq3-run.json +2c663909db5624832bae8393648a1c50a4ac00ab580f4ae7a5f8bdbdce0ef3e2 rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r11/rq3-run.json +5a663ef49d5860187fd2a7fc0453ae2f3e3e312ff1224cb75db57c3eb498711a rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r12/rq3-run.json +2af0bd978aa2b92842b5308ec262f508d3f244b9640cc7f16c0dd5f16c05f843 rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r13/rq3-run.json +77331f8a9041755c77155da29b8176b4c27b674d741f23ee4bf213f84e008949 rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r14/rq3-run.json +2f4b249f9afba300094bb2ce3b50265abe09350ce8959ec2b471ece8ffdfa6ae rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r15/rq3-run.json +0db6e249bf8b6ac0d6d255437a9f868d87641f66fc7fb93f7ca809674884bdb7 rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r16/rq3-run.json +76aac9803ac18a5f6f8f0016ba3b7970fcff475a8475fc959834a4a88f1a0442 rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r17/rq3-run.json +2a25407358b9ab36cb4997af9f232dcb1f0247015268278ec63cada29cfe6a39 rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r18/rq3-run.json +e0f404e78b35404f789120a8b898e03a6ca1bad292bf1ac4efc68bd2004188f0 rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r19/rq3-run.json +2515a88a3116a0d490b9d5484517479ed18af45436b33ffa31ee1293ee7fb6e8 rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r2/rq3-run.json +73593f9667e94ed16fdfa6b943d4b2cc7174cd62ca2cf6ec25f5bc4354d3ec62 rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r20/rq3-run.json +9d89871c47cc12ac826dbeecab7d3943bff252248ecbbc5e88e444ec83f1fda1 rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r21/rq3-run.json +d6753dc2097d22c67cc31fdfca225b336e8439a4975b4f1428879796c1d0410e rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r22/rq3-run.json +befe39c32b87e73169ec6c111ca79f15fbb24d4952b5dd817ce766399e10824e rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r23/rq3-run.json +cad75228e7ef404fbc54561d3f5b4969f441b047abaa0663ae1d51322f43acaa rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r24/rq3-run.json +2817c84be3af073fa471e0287a845d8147d01a1d0f8e800faa281ce02885afe4 rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r25/rq3-run.json +3e7d841db693adb2fe61218588da3e269d31e337242241d0ac87115f516d221c rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r26/rq3-run.json +6a5b5d07bcdf363f4fb959135de6cc8b23c48e5a76888abd0fd2e67e80e5673f rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r27/rq3-run.json +44d512c8eeef83c410399303d482ef760fa490630b26a4cc5ca7294e7f324a14 rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r28/rq3-run.json +6ea17d94630ccc117637b3a2ea0ddc8926f0085ad7cd5cea2aeba8de66997150 rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r29/rq3-run.json +1d07bbd03e31d2717103d5d766d20c43730e3d65d1b95e551531f198744544cb rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r3/rq3-run.json +a22cd7af4000eea7529ba1e7dd257184f702d77fed44965a9a91c7944b65c2c9 rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r4/rq3-run.json +7169dd2d4cf99c53537ce9fd5608f7dda118f645d01cbee5124721f3d03654cc rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r5/rq3-run.json +3c6352374e680178deb5f52f20839202cebf8536fe3b1afb7e0a8e475ae36c27 rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r6/rq3-run.json +4cb9c0fcf183a53bb7dc428c12ba3482adae9d7554238e49520b4848b7599fb8 rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r7/rq3-run.json +763e75f8b465cf8b2b41c644b47f9e2c71a85995c569c2700e929feecd087c96 rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r8/rq3-run.json +3bdc4dcf22de5f09f033b03bd16c01cb9c1cd4fc210ce9b57c8a0c65ec663e80 rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r9/rq3-run.json +257326c8a34c602e028fca9593e8fe628a950eff63138f6eeeb425959112ed5a rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r0/rq3-run.json +1694018a795e64d07b5ff83f5ec512a0c6e38a4f210b3ea1d54b41281d54e21f rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r1/rq3-run.json +bdfb55a3b606ef441237ed0828d504dd885d6730b0de01b0d3cc50f8a5c05826 rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r10/rq3-run.json +3a13e1de470c79965972a275d72e4c8bf8d1195e3a55a896520904de4d6c2e68 rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r11/rq3-run.json +4449a0a53f7033cb7befd8520501b0f00b9201738e517c023576d7b861720935 rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r12/rq3-run.json +45105408efac8d96b449dd3e1945f5727f9f0189a85d3e6bfa5a3c2db0b43882 rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r13/rq3-run.json +07a9e31fec1807d5200978cbad6267966acd74d3eec640868e87bb778d4d4722 rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r14/rq3-run.json +76830fb704f7439be2443fb2c051aeaeaaafddf32e35f209549bf74aab4ffe82 rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r15/rq3-run.json +a4d9a9eb631a3c2c459252f79d0ac65c743440cef445c2545f7e08141130b34e rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r16/rq3-run.json +7357f8e01e000bea1c27a0ccaed7d41f01a5e1f6f75569561a6e1901dc696442 rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r17/rq3-run.json +4ee979e846426d382f6aea04e205cfe33a3478add09273956decf5aa6db28c37 rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r18/rq3-run.json +19e6c0d747dd802e324ca6225a56e4a849e7db42f09df3b61c703aad4adb7082 rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r19/rq3-run.json +cd2638c191fe0a73797f3e2eefbcfba8af9bca0759e6473c420e8af523d2c99e rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r2/rq3-run.json +9f3730ef5c8feeed3c2578e8a30e471eeabc5c98d2a57514ad053d6b9328cb36 rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r20/rq3-run.json +f947eec0f931023d0e6848809ee3484df191099236c50db46a32d3fcaecf88d8 rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r21/rq3-run.json +9d5bc5dff8954fb6641ae9d2f9f83d659339867ac7395278c88d0e190991cd92 rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r22/rq3-run.json +1cbeaf4904aff435ac72d8e002bd909ceeceeda2c176794f959b8c7264b021c0 rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r23/rq3-run.json +efaaad8ffd17cfe3c55e62d09f5b1dfd670c2474ea6219931df8e08eef2d83a0 rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r24/rq3-run.json +03c89c522e5e2cce9181c20f62edd434a734d8dc530629b36bfb6b5e58173c50 rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r25/rq3-run.json +5c171990a45aca7adede446a2bb8b5cec5b72d454d5f4ae061b0883ec80eb5ab rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r26/rq3-run.json +4762af92e07ba3f8c7e6f0ffecc25af15469cbb3d15e151a9acb12619cf37e7e rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r27/rq3-run.json +ec29c89ef5e42ec618ed848c23f9fa35709ce663053929214f05226a055ae2d1 rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r28/rq3-run.json +68f3441aff941c9aabba81321e9352b966bf0883a1afb6b05172b4d5ea3c32d9 rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r29/rq3-run.json +d8d6aaaa09b62f293750d8f46422ae27e504fd8863591f3f1c34f1a2135c6b0c rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r3/rq3-run.json +20e794c0082d6ec4ebdee3f8e659b5f6a19280a0706ef9910dc5af62329bbe32 rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r4/rq3-run.json +5346c57b2b4822aa36906b4348d9be1c4c0c6504cd91676c49af5fa4c637769c rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r5/rq3-run.json +375089aa2ffa966668ad9063acb37e21f3ad176996bdd8f5a66f20d2bca7da53 rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r6/rq3-run.json +8e5f07eb6b52125ccb13a58d4fd2316075a416670fe2391627c0954a42bf36f0 rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r7/rq3-run.json +e334641cd823338129c050a35a61d1d47a50976a6110301a42294a2e80b47c4a rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r8/rq3-run.json +e2c48513c108020c7e29a3107c7f8c5bfdb8bb8374120fc1827177d81fe772a1 rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r9/rq3-run.json diff --git a/output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-battery-results.json b/output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-battery-results.json new file mode 100644 index 0000000..aab1bf8 --- /dev/null +++ b/output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-battery-results.json @@ -0,0 +1,3415 @@ +{ + "c_circuits": 50, + "cells": { + "RQ3/topo=1house/bridge=off/selector=agent/churn=kp30s20": { + "added_latency_ms": [ + 8837.660030461848, + 8861.835315357894, + 8668.947615893558, + 8623.93062422052, + 10057.20242462121, + 8808.744275011122, + 8784.593624528497, + 8868.973311036825, + 8879.108164459467, + 8783.528310246766, + 8867.379901465029, + 8799.000394064933, + 8851.78358736448, + 8852.487781550735, + 8901.02440933697, + 8934.026485076174, + 8952.230052324012, + 8882.415926782414, + 8930.932803545147, + 8931.55911238864, + 8893.849051557481, + 8894.999108510092, + 8808.362854178995, + 8821.967675583437, + 8808.459802763537, + 8958.432310959324, + 8997.983438894153, + 8825.715195154771, + 8852.76440391317, + 9030.908505665138 + ], + "runs": 30, + "strategy": "agent", + "throughput_retention": [ + 1.0, + 0.9444444444444444, + 1.0, + 1.0, + 0.9545454545454546, + 0.9411764705882353, + 1.0, + 1.0, + 1.0, + 1.0, + 0.95, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.9444444444444444, + 1.0, + 1.0, + 1.0, + 1.0, + 0.9375, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0 + ] + }, + "RQ3/topo=1house/bridge=off/selector=random/churn=kp30s20": { + "added_latency_ms": [ + 8807.099595665932, + 8877.263363217935, + 9001.896020723507, + 8934.747805120423, + 8864.67396048829, + 8847.504741046578, + 8898.065702524036, + 8871.30873883143, + 8785.199264064431, + 8993.708913447335, + 8986.649100901559, + 8920.820370782167, + 8927.070990204811, + 8808.025279082358, + 8805.704248603433, + 8820.992421591654, + 8879.3747627642, + 8827.547430060804, + 8938.87149891816, + 8884.300630772486, + 8903.115424560383, + 8811.059419764206, + 8968.811745988205, + 8874.944143230096, + 8909.448232967407, + 8878.934999462217, + 8781.497995601967, + 8741.718172095716, + 8777.821063064039, + 8933.335005072877 + ], + "runs": 30, + "strategy": "random", + "throughput_retention": [ + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.9523809523809523, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.9090909090909091 + ] + }, + "RQ3/topo=1house/bridge=off/selector=static/churn=kp30s20": { + "added_latency_ms": [ + 9830.769951455295, + 9816.24459195882, + 9828.50203034468, + 9667.497479589656, + 8919.178417418152, + 9384.677413385361, + 8887.683250475675, + 9084.53763811849, + 11634.897468611598, + 9001.072274055332, + 8678.027565358207, + 8906.375291990116, + 8846.016283379868, + 8902.138963807374, + 8863.950844388455, + 8857.28882602416, + 8833.572200732306, + 8878.94602608867, + 8805.525227449834, + 8826.84023748152, + 8845.379926497117, + 8812.783573521301, + 8893.704002490267, + 8995.345027418807, + 8873.488415032625, + 9099.247937323526, + 8994.237255770713, + 8912.992535624653, + 8843.840296845883, + 8901.263706153259 + ], + "runs": 30, + "strategy": "static", + "throughput_retention": [ + 1.0, + 1.0, + 1.0, + 1.0, + 0.9090909090909091, + 1.0, + 1.0, + 0.9545454545454546, + 1.0, + 1.0, + 0.9473684210526315, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.9375, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.9411764705882353, + 1.0, + 1.0, + 1.0 + ] + } + }, + "engine": "docker", + "hops": 3, + "measured_from": "live-docker-e2e", + "n_runs": 90, + "order_seed": 20260719, + "pool_size": 8, + "r_runs": 30, + "runs": [ + { + "added_latency_ms": 9830.769951455295, + "cell_id": "RQ3/topo=1house/bridge=off/selector=static/churn=kp30s20", + "deferred": 0, + "drops": 20, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 0.0, + 0.0, + 1.0, + 2.0, + 0.0, + 1.0, + 2.0, + 0.0, + 0.0, + 1.0, + 0.0, + 1.0, + 1.0, + 1.0, + 0.0, + 1.0, + 0.0, + 1.0, + 0.0 + ], + "rebuilds": 20, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r22", + "run_index": 22, + "seed": 16087784176769436461, + "steps": 20, + "strategy": "static", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 9816.24459195882, + "cell_id": "RQ3/topo=1house/bridge=off/selector=static/churn=kp30s20", + "deferred": 0, + "drops": 15, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 0.0, + 2.0, + 0.0, + 0.0, + 2.0, + 1.0, + 0.0, + 0.0, + 6.0, + 3.0, + 2.0, + 0.0, + 1.0, + 2.0 + ], + "rebuilds": 15, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r1", + "run_index": 1, + "seed": 6630313967092928729, + "steps": 20, + "strategy": "static", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 9828.50203034468, + "cell_id": "RQ3/topo=1house/bridge=off/selector=static/churn=kp30s20", + "deferred": 0, + "drops": 14, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 2.0, + 0.0, + 0.0, + 1.0, + 2.0, + 1.0, + 0.0, + 0.0, + 1.0, + 2.0, + 0.0, + 2.0, + 5.0 + ], + "rebuilds": 14, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r24", + "run_index": 24, + "seed": 13515935951670286305, + "steps": 20, + "strategy": "static", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 9667.497479589656, + "cell_id": "RQ3/topo=1house/bridge=off/selector=static/churn=kp30s20", + "deferred": 0, + "drops": 16, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 0.0, + 2.0, + 2.0, + 1.0, + 2.0, + 0.0, + 2.0, + 3.0, + 0.0, + 0.0, + 2.0, + 0.0, + 1.0, + 1.0, + 1.0 + ], + "rebuilds": 16, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r20", + "run_index": 20, + "seed": 17371661291353427865, + "steps": 20, + "strategy": "static", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8919.178417418152, + "cell_id": "RQ3/topo=1house/bridge=off/selector=static/churn=kp30s20", + "deferred": 1, + "drops": 11, + "every_drop_rebuilt": false, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 2.0, + 4.0, + 3.0, + 1.0, + 2.0, + 0.0, + 1.0, + 1.0, + 1.0 + ], + "rebuilds": 10, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r8", + "run_index": 8, + "seed": 10011143392304782265, + "steps": 20, + "strategy": "static", + "throughput_retention": 0.9090909090909091 + }, + { + "added_latency_ms": 9384.677413385361, + "cell_id": "RQ3/topo=1house/bridge=off/selector=static/churn=kp30s20", + "deferred": 0, + "drops": 13, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 2.0, + 0.0, + 2.0, + 3.0, + 2.0, + 2.0, + 0.0, + 2.0, + 1.0, + 1.0, + 2.0, + 1.0 + ], + "rebuilds": 13, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r6", + "run_index": 6, + "seed": 836667626899191429, + "steps": 20, + "strategy": "static", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8887.683250475675, + "cell_id": "RQ3/topo=1house/bridge=off/selector=static/churn=kp30s20", + "deferred": 0, + "drops": 20, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 0.0, + 1.0, + 1.0, + 0.0, + 1.0, + 1.0, + 2.0, + 0.0, + 0.0, + 0.0, + 1.0, + 1.0, + 1.0, + 1.0, + 4.0, + 1.0, + 2.0, + 0.0, + 1.0 + ], + "rebuilds": 20, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r18", + "run_index": 18, + "seed": 15253112393218679906, + "steps": 20, + "strategy": "static", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 9084.53763811849, + "cell_id": "RQ3/topo=1house/bridge=off/selector=static/churn=kp30s20", + "deferred": 1, + "drops": 22, + "every_drop_rebuilt": false, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 0.0, + 0.0, + 2.0, + 1.0, + 0.0, + 1.0, + 0.0, + 1.0, + 3.0, + 1.0, + 1.0, + 0.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.0, + 1.0, + 0.0, + 2.0 + ], + "rebuilds": 21, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r28", + "run_index": 28, + "seed": 17687245211475078502, + "steps": 20, + "strategy": "static", + "throughput_retention": 0.9545454545454546 + }, + { + "added_latency_ms": 11634.897468611598, + "cell_id": "RQ3/topo=1house/bridge=off/selector=static/churn=kp30s20", + "deferred": 0, + "drops": 19, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 0.0, + 1.0, + 2.0, + 0.0, + 2.0, + 0.0, + 2.0, + 1.0, + 1.0, + 0.0, + 3.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.0, + 1.0, + 1.0 + ], + "rebuilds": 19, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r16", + "run_index": 16, + "seed": 18277630771554192698, + "steps": 20, + "strategy": "static", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 9001.072274055332, + "cell_id": "RQ3/topo=1house/bridge=off/selector=static/churn=kp30s20", + "deferred": 0, + "drops": 18, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 0.0, + 0.0, + 1.0, + 1.0, + 3.0, + 0.0, + 2.0, + 1.0, + 0.0, + 2.0, + 1.0, + 1.0, + 1.0, + 0.0, + 1.0, + 1.0, + 4.0 + ], + "rebuilds": 18, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r0", + "run_index": 0, + "seed": 9367372173334061740, + "steps": 20, + "strategy": "static", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8678.027565358207, + "cell_id": "RQ3/topo=1house/bridge=off/selector=static/churn=kp30s20", + "deferred": 1, + "drops": 19, + "every_drop_rebuilt": false, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 3.0, + 2.0, + 1.0, + 0.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.0, + 2.0, + 1.0, + 1.0, + 0.0, + 1.0, + 1.0, + 0.0, + 1.0 + ], + "rebuilds": 18, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r14", + "run_index": 14, + "seed": 12634594441809140906, + "steps": 20, + "strategy": "static", + "throughput_retention": 0.9473684210526315 + }, + { + "added_latency_ms": 8906.375291990116, + "cell_id": "RQ3/topo=1house/bridge=off/selector=static/churn=kp30s20", + "deferred": 0, + "drops": 27, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 0.0, + 0.0, + 0.0, + 2.0, + 1.0, + 0.0, + 1.0, + 1.0, + 0.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.0, + 0.0, + 3.0, + 1.0, + 1.0, + 0.0, + 1.0, + 1.0, + 0.0, + 2.0, + 1.0, + 0.0, + 0.0 + ], + "rebuilds": 27, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r7", + "run_index": 7, + "seed": 14188752224994722441, + "steps": 20, + "strategy": "static", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8846.016283379868, + "cell_id": "RQ3/topo=1house/bridge=off/selector=static/churn=kp30s20", + "deferred": 0, + "drops": 19, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 1.0, + 1.0, + 1.0, + 0.0, + 1.0, + 3.0, + 1.0, + 0.0, + 1.0, + 1.0, + 0.0, + 1.0, + 0.0, + 2.0, + 0.0, + 2.0, + 1.0, + 0.0 + ], + "rebuilds": 19, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r3", + "run_index": 3, + "seed": 11208649205148287663, + "steps": 20, + "strategy": "static", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8902.138963807374, + "cell_id": "RQ3/topo=1house/bridge=off/selector=static/churn=kp30s20", + "deferred": 0, + "drops": 18, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 0.0, + 1.0, + 0.0, + 2.0, + 1.0, + 1.0, + 2.0, + 0.0, + 2.0, + 2.0, + 0.0, + 2.0, + 0.0, + 3.0, + 1.0, + 0.0, + 1.0 + ], + "rebuilds": 18, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r25", + "run_index": 25, + "seed": 11291892362849214784, + "steps": 20, + "strategy": "static", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8863.950844388455, + "cell_id": "RQ3/topo=1house/bridge=off/selector=static/churn=kp30s20", + "deferred": 0, + "drops": 19, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 1.0, + 0.0, + 0.0, + 2.0, + 0.0, + 1.0, + 1.0, + 1.0, + 0.0, + 1.0, + 1.0, + 0.0, + 1.0, + 0.0, + 1.0, + 1.0, + 1.0, + 4.0 + ], + "rebuilds": 19, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r2", + "run_index": 2, + "seed": 3976366511193135127, + "steps": 20, + "strategy": "static", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8837.660030461848, + "cell_id": "RQ3/topo=1house/bridge=off/selector=agent/churn=kp30s20", + "deferred": 0, + "drops": 22, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 0.0, + 1.0, + 0.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 2.0, + 1.0, + 1.0, + 1.0, + 0.0, + 2.0, + 0.0, + 1.0, + 0.0, + 3.0, + 0.0, + 1.0 + ], + "rebuilds": 22, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r12", + "run_index": 12, + "seed": 14158303034302257116, + "steps": 20, + "strategy": "agent", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8807.099595665932, + "cell_id": "RQ3/topo=1house/bridge=off/selector=random/churn=kp30s20", + "deferred": 0, + "drops": 12, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 1.0, + 2.0, + 2.0, + 2.0, + 0.0, + 3.0, + 2.0, + 0.0, + 2.0, + 1.0, + 4.0 + ], + "rebuilds": 12, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r21", + "run_index": 21, + "seed": 8178898065691935613, + "steps": 20, + "strategy": "random", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8877.263363217935, + "cell_id": "RQ3/topo=1house/bridge=off/selector=random/churn=kp30s20", + "deferred": 0, + "drops": 14, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 6.0, + 3.0, + 0.0, + 1.0, + 1.0, + 0.0, + 0.0, + 1.0, + 3.0, + 1.0, + 1.0, + 1.0, + 1.0 + ], + "rebuilds": 14, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r11", + "run_index": 11, + "seed": 14229720496423468076, + "steps": 20, + "strategy": "random", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8861.835315357894, + "cell_id": "RQ3/topo=1house/bridge=off/selector=agent/churn=kp30s20", + "deferred": 1, + "drops": 18, + "every_drop_rebuilt": false, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 1.0, + 0.0, + 0.0, + 1.0, + 2.0, + 0.0, + 2.0, + 2.0, + 0.0, + 1.0, + 1.0, + 3.0, + 1.0, + 2.0, + 0.0, + 1.0 + ], + "rebuilds": 17, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r23", + "run_index": 23, + "seed": 6178346823415373913, + "steps": 20, + "strategy": "agent", + "throughput_retention": 0.9444444444444444 + }, + { + "added_latency_ms": 9001.896020723507, + "cell_id": "RQ3/topo=1house/bridge=off/selector=random/churn=kp30s20", + "deferred": 0, + "drops": 14, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 3.0, + 0.0, + 1.0, + 1.0, + 1.0, + 0.0, + 1.0, + 3.0, + 3.0, + 1.0, + 1.0, + 3.0, + 1.0 + ], + "rebuilds": 14, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r10", + "run_index": 10, + "seed": 16960546821439402222, + "steps": 20, + "strategy": "random", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8934.747805120423, + "cell_id": "RQ3/topo=1house/bridge=off/selector=random/churn=kp30s20", + "deferred": 0, + "drops": 17, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 2.0, + 1.0, + 0.0, + 3.0, + 1.0, + 1.0, + 1.0, + 2.0, + 0.0, + 1.0, + 1.0, + 0.0, + 3.0, + 2.0, + 1.0, + 0.0 + ], + "rebuilds": 17, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r4", + "run_index": 4, + "seed": 16775842078457051607, + "steps": 20, + "strategy": "random", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8864.67396048829, + "cell_id": "RQ3/topo=1house/bridge=off/selector=random/churn=kp30s20", + "deferred": 0, + "drops": 23, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 1.0, + 0.0, + 0.0, + 1.0, + 0.0, + 1.0, + 2.0, + 1.0, + 3.0, + 1.0, + 0.0, + 0.0, + 1.0, + 2.0, + 1.0, + 1.0, + 0.0, + 2.0, + 0.0, + 2.0, + 0.0, + 0.0 + ], + "rebuilds": 23, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r29", + "run_index": 29, + "seed": 7783207762530451041, + "steps": 20, + "strategy": "random", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8668.947615893558, + "cell_id": "RQ3/topo=1house/bridge=off/selector=agent/churn=kp30s20", + "deferred": 0, + "drops": 21, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 0.0, + 0.0, + 0.0, + 1.0, + 1.0, + 2.0, + 1.0, + 0.0, + 2.0, + 1.0, + 0.0, + 1.0, + 1.0, + 1.0, + 0.0, + 4.0, + 0.0, + 1.0, + 2.0, + 1.0 + ], + "rebuilds": 21, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r26", + "run_index": 26, + "seed": 8796116083789013211, + "steps": 20, + "strategy": "agent", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8623.93062422052, + "cell_id": "RQ3/topo=1house/bridge=off/selector=agent/churn=kp30s20", + "deferred": 0, + "drops": 19, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 0.0, + 2.0, + 1.0, + 0.0, + 0.0, + 1.0, + 2.0, + 2.0, + 0.0, + 1.0, + 1.0, + 0.0, + 2.0, + 1.0, + 0.0, + 3.0, + 1.0, + 0.0 + ], + "rebuilds": 19, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r24", + "run_index": 24, + "seed": 10535835380763469014, + "steps": 20, + "strategy": "agent", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8847.504741046578, + "cell_id": "RQ3/topo=1house/bridge=off/selector=random/churn=kp30s20", + "deferred": 0, + "drops": 20, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 2.0, + 0.0, + 1.0, + 0.0, + 1.0, + 3.0, + 1.0, + 0.0, + 2.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 2.0, + 0.0, + 1.0, + 0.0, + 1.0 + ], + "rebuilds": 20, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r12", + "run_index": 12, + "seed": 4439262027348900681, + "steps": 20, + "strategy": "random", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 10057.20242462121, + "cell_id": "RQ3/topo=1house/bridge=off/selector=agent/churn=kp30s20", + "deferred": 1, + "drops": 22, + "every_drop_rebuilt": false, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 0.0, + 0.0, + 3.0, + 0.0, + 3.0, + 0.0, + 1.0, + 2.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.0, + 1.0, + 1.0, + 0.0, + 1.0, + 1.0, + 1.0 + ], + "rebuilds": 21, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r13", + "run_index": 13, + "seed": 6374494410694636325, + "steps": 20, + "strategy": "agent", + "throughput_retention": 0.9545454545454546 + }, + { + "added_latency_ms": 8898.065702524036, + "cell_id": "RQ3/topo=1house/bridge=off/selector=random/churn=kp30s20", + "deferred": 0, + "drops": 19, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 0.0, + 2.0, + 0.0, + 1.0, + 1.0, + 1.0, + 0.0, + 2.0, + 0.0, + 3.0, + 2.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0 + ], + "rebuilds": 19, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r6", + "run_index": 6, + "seed": 3627346239498443444, + "steps": 20, + "strategy": "random", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8808.744275011122, + "cell_id": "RQ3/topo=1house/bridge=off/selector=agent/churn=kp30s20", + "deferred": 1, + "drops": 17, + "every_drop_rebuilt": false, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 0.0, + 1.0, + 1.0, + 2.0, + 1.0, + 2.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.0, + 1.0, + 4.0, + 0.0, + 2.0 + ], + "rebuilds": 16, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r21", + "run_index": 21, + "seed": 17434159462361624673, + "steps": 20, + "strategy": "agent", + "throughput_retention": 0.9411764705882353 + }, + { + "added_latency_ms": 8871.30873883143, + "cell_id": "RQ3/topo=1house/bridge=off/selector=random/churn=kp30s20", + "deferred": 1, + "drops": 21, + "every_drop_rebuilt": false, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.0, + 0.0, + 2.0, + 0.0, + 1.0, + 1.0, + 2.0, + 0.0 + ], + "rebuilds": 20, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r7", + "run_index": 7, + "seed": 14284312584264524989, + "steps": 20, + "strategy": "random", + "throughput_retention": 0.9523809523809523 + }, + { + "added_latency_ms": 8784.593624528497, + "cell_id": "RQ3/topo=1house/bridge=off/selector=agent/churn=kp30s20", + "deferred": 0, + "drops": 24, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 0.0, + 0.0, + 0.0, + 0.0, + 1.0, + 1.0, + 2.0, + 1.0, + 1.0, + 0.0, + 0.0, + 1.0, + 1.0, + 3.0, + 0.0, + 0.0, + 2.0, + 1.0, + 2.0, + 0.0, + 0.0, + 2.0, + 1.0 + ], + "rebuilds": 24, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r25", + "run_index": 25, + "seed": 18017748332809528625, + "steps": 20, + "strategy": "agent", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8785.199264064431, + "cell_id": "RQ3/topo=1house/bridge=off/selector=random/churn=kp30s20", + "deferred": 0, + "drops": 17, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 0.0, + 1.0, + 0.0, + 0.0, + 2.0, + 2.0, + 1.0, + 5.0, + 0.0, + 1.0, + 0.0, + 1.0, + 1.0, + 1.0, + 2.0, + 1.0 + ], + "rebuilds": 17, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r16", + "run_index": 16, + "seed": 3204797884819698204, + "steps": 20, + "strategy": "random", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8868.973311036825, + "cell_id": "RQ3/topo=1house/bridge=off/selector=agent/churn=kp30s20", + "deferred": 0, + "drops": 18, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 0.0, + 1.0, + 1.0, + 2.0, + 1.0, + 3.0, + 1.0, + 0.0, + 2.0, + 3.0, + 0.0, + 0.0, + 1.0, + 1.0, + 0.0, + 1.0, + 1.0 + ], + "rebuilds": 18, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r28", + "run_index": 28, + "seed": 10098418389192135517, + "steps": 20, + "strategy": "agent", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8993.708913447335, + "cell_id": "RQ3/topo=1house/bridge=off/selector=random/churn=kp30s20", + "deferred": 0, + "drops": 17, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 1.0, + 0.0, + 3.0, + 1.0, + 1.0, + 0.0, + 3.0, + 0.0, + 2.0, + 0.0, + 3.0, + 2.0, + 2.0, + 0.0, + 1.0, + 0.0 + ], + "rebuilds": 17, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r2", + "run_index": 2, + "seed": 1739401898737214078, + "steps": 20, + "strategy": "random", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8879.108164459467, + "cell_id": "RQ3/topo=1house/bridge=off/selector=agent/churn=kp30s20", + "deferred": 0, + "drops": 12, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 1.0, + 1.0, + 0.0, + 4.0, + 1.0, + 0.0, + 1.0, + 2.0, + 3.0, + 1.0, + 2.0 + ], + "rebuilds": 12, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r11", + "run_index": 11, + "seed": 15767512560850333077, + "steps": 20, + "strategy": "agent", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8986.649100901559, + "cell_id": "RQ3/topo=1house/bridge=off/selector=random/churn=kp30s20", + "deferred": 0, + "drops": 17, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 4.0, + 1.0, + 1.0, + 0.0, + 0.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.0, + 1.0, + 0.0, + 3.0, + 2.0, + 2.0, + 0.0 + ], + "rebuilds": 17, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r24", + "run_index": 24, + "seed": 15721796934301167956, + "steps": 20, + "strategy": "random", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8920.820370782167, + "cell_id": "RQ3/topo=1house/bridge=off/selector=random/churn=kp30s20", + "deferred": 0, + "drops": 13, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 3.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 3.0, + 1.0, + 1.0, + 0.0, + 1.0, + 2.0 + ], + "rebuilds": 13, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r18", + "run_index": 18, + "seed": 10466670844153308712, + "steps": 20, + "strategy": "random", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8927.070990204811, + "cell_id": "RQ3/topo=1house/bridge=off/selector=random/churn=kp30s20", + "deferred": 0, + "drops": 18, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 1.0, + 0.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.0, + 5.0, + 1.0, + 1.0, + 2.0, + 1.0, + 1.0, + 0.0, + 1.0, + 1.0, + 1.0 + ], + "rebuilds": 18, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r28", + "run_index": 28, + "seed": 9127898839643835018, + "steps": 20, + "strategy": "random", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8783.528310246766, + "cell_id": "RQ3/topo=1house/bridge=off/selector=agent/churn=kp30s20", + "deferred": 0, + "drops": 16, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 0.0, + 2.0, + 1.0, + 1.0, + 2.0, + 1.0, + 0.0, + 1.0, + 2.0, + 0.0, + 1.0, + 2.0, + 2.0, + 3.0, + 1.0 + ], + "rebuilds": 16, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r10", + "run_index": 10, + "seed": 16753480740272410390, + "steps": 20, + "strategy": "agent", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8808.025279082358, + "cell_id": "RQ3/topo=1house/bridge=off/selector=random/churn=kp30s20", + "deferred": 0, + "drops": 16, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 0.0, + 4.0, + 2.0, + 1.0, + 1.0, + 2.0, + 1.0, + 1.0, + 1.0, + 2.0, + 1.0, + 0.0, + 1.0, + 2.0, + 0.0 + ], + "rebuilds": 16, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r19", + "run_index": 19, + "seed": 18068654908094798291, + "steps": 20, + "strategy": "random", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8867.379901465029, + "cell_id": "RQ3/topo=1house/bridge=off/selector=agent/churn=kp30s20", + "deferred": 1, + "drops": 20, + "every_drop_rebuilt": false, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 1.0, + 1.0, + 0.0, + 0.0, + 2.0, + 0.0, + 1.0, + 1.0, + 0.0, + 1.0, + 3.0, + 3.0, + 1.0, + 1.0, + 2.0, + 2.0, + 0.0, + 0.0 + ], + "rebuilds": 19, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r0", + "run_index": 0, + "seed": 16661262391680493370, + "steps": 20, + "strategy": "agent", + "throughput_retention": 0.95 + }, + { + "added_latency_ms": 8805.704248603433, + "cell_id": "RQ3/topo=1house/bridge=off/selector=random/churn=kp30s20", + "deferred": 0, + "drops": 18, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 0.0, + 0.0, + 0.0, + 3.0, + 2.0, + 2.0, + 0.0, + 3.0, + 0.0, + 2.0, + 1.0, + 2.0, + 1.0, + 0.0, + 2.0, + 1.0, + 0.0 + ], + "rebuilds": 18, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r1", + "run_index": 1, + "seed": 11425995555011020922, + "steps": 20, + "strategy": "random", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8799.000394064933, + "cell_id": "RQ3/topo=1house/bridge=off/selector=agent/churn=kp30s20", + "deferred": 0, + "drops": 13, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 2.0, + 2.0, + 2.0, + 0.0, + 1.0, + 3.0, + 1.0, + 1.0, + 2.0, + 3.0, + 1.0, + 0.0 + ], + "rebuilds": 13, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r5", + "run_index": 5, + "seed": 3213893795518785560, + "steps": 20, + "strategy": "agent", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8851.78358736448, + "cell_id": "RQ3/topo=1house/bridge=off/selector=agent/churn=kp30s20", + "deferred": 0, + "drops": 23, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 0.0, + 0.0, + 1.0, + 0.0, + 2.0, + 2.0, + 0.0, + 1.0, + 1.0, + 0.0, + 1.0, + 0.0, + 2.0, + 0.0, + 1.0, + 1.0, + 2.0, + 0.0, + 0.0, + 1.0, + 2.0, + 1.0 + ], + "rebuilds": 23, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r14", + "run_index": 14, + "seed": 8917414950898246867, + "steps": 20, + "strategy": "agent", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8820.992421591654, + "cell_id": "RQ3/topo=1house/bridge=off/selector=random/churn=kp30s20", + "deferred": 0, + "drops": 17, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 0.0, + 1.0, + 0.0, + 1.0, + 2.0, + 0.0, + 3.0, + 4.0, + 1.0, + 1.0, + 1.0, + 0.0, + 2.0, + 2.0, + 0.0, + 1.0 + ], + "rebuilds": 17, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r27", + "run_index": 27, + "seed": 331021859493055528, + "steps": 20, + "strategy": "random", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8852.487781550735, + "cell_id": "RQ3/topo=1house/bridge=off/selector=agent/churn=kp30s20", + "deferred": 0, + "drops": 18, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 2.0, + 1.0, + 1.0, + 2.0, + 2.0, + 0.0, + 2.0, + 0.0, + 0.0, + 1.0, + 1.0, + 2.0, + 2.0, + 2.0, + 0.0, + 1.0, + 0.0 + ], + "rebuilds": 18, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r19", + "run_index": 19, + "seed": 17607509839346677083, + "steps": 20, + "strategy": "agent", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8879.3747627642, + "cell_id": "RQ3/topo=1house/bridge=off/selector=random/churn=kp30s20", + "deferred": 0, + "drops": 21, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 1.0, + 0.0, + 0.0, + 2.0, + 1.0, + 0.0, + 2.0, + 2.0, + 1.0, + 0.0, + 1.0, + 1.0, + 0.0, + 3.0, + 2.0, + 1.0, + 0.0, + 0.0, + 1.0, + 1.0 + ], + "rebuilds": 21, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r3", + "run_index": 3, + "seed": 10316495177465632795, + "steps": 20, + "strategy": "random", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8827.547430060804, + "cell_id": "RQ3/topo=1house/bridge=off/selector=random/churn=kp30s20", + "deferred": 0, + "drops": 18, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 2.0, + 1.0, + 0.0, + 4.0, + 0.0, + 0.0, + 0.0, + 2.0, + 2.0, + 2.0, + 0.0, + 1.0, + 0.0, + 1.0, + 1.0, + 0.0, + 1.0 + ], + "rebuilds": 18, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r17", + "run_index": 17, + "seed": 10532807285506099109, + "steps": 20, + "strategy": "random", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8901.02440933697, + "cell_id": "RQ3/topo=1house/bridge=off/selector=agent/churn=kp30s20", + "deferred": 0, + "drops": 14, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 2.0, + 0.0, + 0.0, + 1.0, + 1.0, + 2.0, + 0.0, + 1.0, + 1.0, + 1.0, + 6.0, + 1.0, + 2.0 + ], + "rebuilds": 14, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r2", + "run_index": 2, + "seed": 16251376946002760400, + "steps": 20, + "strategy": "agent", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8934.026485076174, + "cell_id": "RQ3/topo=1house/bridge=off/selector=agent/churn=kp30s20", + "deferred": 0, + "drops": 17, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 1.0, + 0.0, + 2.0, + 1.0, + 1.0, + 1.0, + 0.0, + 0.0, + 1.0, + 1.0, + 3.0, + 4.0, + 2.0, + 1.0, + 0.0, + 0.0 + ], + "rebuilds": 17, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r9", + "run_index": 9, + "seed": 1321886396185688516, + "steps": 20, + "strategy": "agent", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8938.87149891816, + "cell_id": "RQ3/topo=1house/bridge=off/selector=random/churn=kp30s20", + "deferred": 0, + "drops": 13, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 1.0, + 1.0, + 1.0, + 3.0, + 1.0, + 1.0, + 5.0, + 0.0, + 1.0, + 1.0, + 0.0, + 1.0 + ], + "rebuilds": 13, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r20", + "run_index": 20, + "seed": 3909899859316098184, + "steps": 20, + "strategy": "random", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8884.300630772486, + "cell_id": "RQ3/topo=1house/bridge=off/selector=random/churn=kp30s20", + "deferred": 0, + "drops": 16, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 3.0, + 2.0, + 3.0, + 0.0, + 2.0, + 0.0, + 1.0, + 1.0, + 2.0, + 0.0, + 0.0, + 1.0, + 1.0, + 1.0, + 1.0 + ], + "rebuilds": 16, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r5", + "run_index": 5, + "seed": 8182447014304330466, + "steps": 20, + "strategy": "random", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8952.230052324012, + "cell_id": "RQ3/topo=1house/bridge=off/selector=agent/churn=kp30s20", + "deferred": 1, + "drops": 18, + "every_drop_rebuilt": false, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 2.0, + 1.0, + 1.0, + 0.0, + 1.0, + 3.0, + 0.0, + 2.0, + 1.0, + 1.0, + 1.0, + 0.0, + 1.0, + 0.0, + 2.0, + 3.0 + ], + "rebuilds": 17, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r6", + "run_index": 6, + "seed": 8880208142306762226, + "steps": 20, + "strategy": "agent", + "throughput_retention": 0.9444444444444444 + }, + { + "added_latency_ms": 8882.415926782414, + "cell_id": "RQ3/topo=1house/bridge=off/selector=agent/churn=kp30s20", + "deferred": 0, + "drops": 17, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 0.0, + 0.0, + 1.0, + 1.0, + 1.0, + 1.0, + 2.0, + 0.0, + 4.0, + 0.0, + 1.0, + 1.0, + 0.0, + 5.0, + 0.0, + 1.0 + ], + "rebuilds": 17, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r4", + "run_index": 4, + "seed": 15667670401208486990, + "steps": 20, + "strategy": "agent", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8903.115424560383, + "cell_id": "RQ3/topo=1house/bridge=off/selector=random/churn=kp30s20", + "deferred": 0, + "drops": 18, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 0.0, + 1.0, + 1.0, + 2.0, + 0.0, + 2.0, + 1.0, + 1.0, + 3.0, + 0.0, + 2.0, + 0.0, + 0.0, + 1.0, + 0.0, + 1.0, + 1.0 + ], + "rebuilds": 18, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r26", + "run_index": 26, + "seed": 10203686528384801371, + "steps": 20, + "strategy": "random", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8811.059419764206, + "cell_id": "RQ3/topo=1house/bridge=off/selector=random/churn=kp30s20", + "deferred": 0, + "drops": 21, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 0.0, + 0.0, + 1.0, + 3.0, + 1.0, + 0.0, + 1.0, + 1.0, + 1.0, + 0.0, + 2.0, + 2.0, + 1.0, + 2.0, + 1.0, + 0.0, + 1.0, + 1.0, + 1.0, + 0.0 + ], + "rebuilds": 21, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r13", + "run_index": 13, + "seed": 8202396444355219667, + "steps": 20, + "strategy": "random", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8968.811745988205, + "cell_id": "RQ3/topo=1house/bridge=off/selector=random/churn=kp30s20", + "deferred": 0, + "drops": 17, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 0.0, + 0.0, + 3.0, + 0.0, + 1.0, + 3.0, + 0.0, + 2.0, + 0.0, + 1.0, + 2.0, + 1.0, + 1.0, + 1.0, + 4.0, + 0.0 + ], + "rebuilds": 17, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r25", + "run_index": 25, + "seed": 18043627777715273245, + "steps": 20, + "strategy": "random", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8930.932803545147, + "cell_id": "RQ3/topo=1house/bridge=off/selector=agent/churn=kp30s20", + "deferred": 0, + "drops": 21, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 0.0, + 0.0, + 1.0, + 3.0, + 1.0, + 3.0, + 1.0, + 0.0, + 0.0, + 1.0, + 0.0, + 1.0, + 1.0, + 2.0, + 1.0, + 0.0, + 1.0, + 1.0, + 0.0, + 1.0 + ], + "rebuilds": 21, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r29", + "run_index": 29, + "seed": 8210726026505286158, + "steps": 20, + "strategy": "agent", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8931.55911238864, + "cell_id": "RQ3/topo=1house/bridge=off/selector=agent/churn=kp30s20", + "deferred": 0, + "drops": 21, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 1.0, + 0.0, + 0.0, + 2.0, + 1.0, + 3.0, + 1.0, + 1.0, + 0.0, + 0.0, + 0.0, + 3.0, + 0.0, + 1.0, + 1.0, + 1.0, + 0.0, + 2.0, + 0.0, + 1.0 + ], + "rebuilds": 21, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r17", + "run_index": 17, + "seed": 193293672290395355, + "steps": 20, + "strategy": "agent", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8874.944143230096, + "cell_id": "RQ3/topo=1house/bridge=off/selector=random/churn=kp30s20", + "deferred": 0, + "drops": 19, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 2.0, + 1.0, + 0.0, + 2.0, + 1.0, + 2.0, + 0.0, + 1.0, + 2.0, + 1.0, + 0.0, + 3.0, + 0.0, + 0.0, + 1.0, + 1.0, + 0.0, + 2.0 + ], + "rebuilds": 19, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r14", + "run_index": 14, + "seed": 764805252441877217, + "steps": 20, + "strategy": "random", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8893.849051557481, + "cell_id": "RQ3/topo=1house/bridge=off/selector=agent/churn=kp30s20", + "deferred": 0, + "drops": 23, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 1.0, + 0.0, + 0.0, + 0.0, + 1.0, + 1.0, + 2.0, + 1.0, + 1.0, + 0.0, + 1.0, + 0.0, + 1.0, + 0.0, + 2.0, + 1.0, + 0.0, + 3.0, + 1.0, + 1.0, + 0.0, + 1.0 + ], + "rebuilds": 23, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r27", + "run_index": 27, + "seed": 3706147200606870934, + "steps": 20, + "strategy": "agent", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8894.999108510092, + "cell_id": "RQ3/topo=1house/bridge=off/selector=agent/churn=kp30s20", + "deferred": 1, + "drops": 16, + "every_drop_rebuilt": false, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 1.0, + 2.0, + 1.0, + 2.0, + 0.0, + 1.0, + 1.0, + 1.0, + 1.0, + 2.0, + 1.0, + 0.0, + 4.0, + 0.0 + ], + "rebuilds": 15, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r15", + "run_index": 15, + "seed": 10043955635765470195, + "steps": 20, + "strategy": "agent", + "throughput_retention": 0.9375 + }, + { + "added_latency_ms": 8808.362854178995, + "cell_id": "RQ3/topo=1house/bridge=off/selector=agent/churn=kp30s20", + "deferred": 0, + "drops": 14, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 1.0, + 3.0, + 2.0, + 0.0, + 2.0, + 1.0, + 1.0, + 1.0, + 0.0, + 2.0, + 3.0, + 0.0, + 1.0 + ], + "rebuilds": 14, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r16", + "run_index": 16, + "seed": 7021063862975976219, + "steps": 20, + "strategy": "agent", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8909.448232967407, + "cell_id": "RQ3/topo=1house/bridge=off/selector=random/churn=kp30s20", + "deferred": 0, + "drops": 22, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.0, + 0.0, + 1.0, + 1.0, + 1.0, + 0.0, + 1.0, + 1.0, + 0.0, + 2.0, + 2.0, + 3.0, + 0.0, + 0.0, + 1.0 + ], + "rebuilds": 22, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r22", + "run_index": 22, + "seed": 3674225224219430868, + "steps": 20, + "strategy": "random", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8821.967675583437, + "cell_id": "RQ3/topo=1house/bridge=off/selector=agent/churn=kp30s20", + "deferred": 0, + "drops": 19, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 0.0, + 1.0, + 1.0, + 0.0, + 1.0, + 1.0, + 1.0, + 0.0, + 1.0, + 0.0, + 2.0, + 1.0, + 1.0, + 1.0, + 3.0, + 1.0, + 2.0, + 2.0 + ], + "rebuilds": 19, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r20", + "run_index": 20, + "seed": 1227328312346801546, + "steps": 20, + "strategy": "agent", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8878.934999462217, + "cell_id": "RQ3/topo=1house/bridge=off/selector=random/churn=kp30s20", + "deferred": 0, + "drops": 13, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 0.0, + 1.0, + 3.0, + 0.0, + 1.0, + 1.0, + 1.0, + 2.0, + 3.0, + 4.0, + 3.0, + 0.0 + ], + "rebuilds": 13, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r15", + "run_index": 15, + "seed": 17450616223277228082, + "steps": 20, + "strategy": "random", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8808.459802763537, + "cell_id": "RQ3/topo=1house/bridge=off/selector=agent/churn=kp30s20", + "deferred": 0, + "drops": 22, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 1.0, + 0.0, + 2.0, + 0.0, + 1.0, + 0.0, + 1.0, + 1.0, + 1.0, + 2.0, + 1.0, + 0.0, + 3.0, + 0.0, + 0.0, + 2.0, + 0.0, + 0.0, + 3.0, + 0.0, + 0.0 + ], + "rebuilds": 22, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r18", + "run_index": 18, + "seed": 3301733400940646521, + "steps": 20, + "strategy": "agent", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8958.432310959324, + "cell_id": "RQ3/topo=1house/bridge=off/selector=agent/churn=kp30s20", + "deferred": 0, + "drops": 21, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 1.0, + 2.0, + 1.0, + 0.0, + 1.0, + 2.0, + 0.0, + 1.0, + 1.0, + 2.0, + 1.0, + 1.0, + 1.0, + 0.0, + 1.0, + 0.0, + 1.0, + 2.0, + 0.0, + 1.0 + ], + "rebuilds": 21, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r7", + "run_index": 7, + "seed": 12045556387598553594, + "steps": 20, + "strategy": "agent", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8781.497995601967, + "cell_id": "RQ3/topo=1house/bridge=off/selector=random/churn=kp30s20", + "deferred": 0, + "drops": 22, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 1.0, + 1.0, + 1.0, + 2.0, + 0.0, + 1.0, + 2.0, + 0.0, + 1.0, + 1.0, + 0.0, + 0.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 2.0, + 1.0, + 1.0, + 0.0 + ], + "rebuilds": 22, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r0", + "run_index": 0, + "seed": 4296170923443173876, + "steps": 20, + "strategy": "random", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8997.983438894153, + "cell_id": "RQ3/topo=1house/bridge=off/selector=agent/churn=kp30s20", + "deferred": 0, + "drops": 22, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 2.0, + 1.0, + 0.0, + 2.0, + 0.0, + 1.0, + 0.0, + 1.0, + 0.0, + 2.0, + 0.0, + 1.0, + 0.0, + 2.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.0, + 0.0, + 2.0 + ], + "rebuilds": 22, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r1", + "run_index": 1, + "seed": 4067513711098866379, + "steps": 20, + "strategy": "agent", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8741.718172095716, + "cell_id": "RQ3/topo=1house/bridge=off/selector=random/churn=kp30s20", + "deferred": 0, + "drops": 21, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 0.0, + 2.0, + 1.0, + 5.0, + 1.0, + 0.0, + 2.0, + 1.0, + 0.0, + 1.0, + 0.0, + 1.0, + 0.0, + 1.0, + 2.0, + 1.0, + 0.0, + 0.0, + 1.0, + 0.0 + ], + "rebuilds": 21, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r23", + "run_index": 23, + "seed": 3190546399781270433, + "steps": 20, + "strategy": "random", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8825.715195154771, + "cell_id": "RQ3/topo=1house/bridge=off/selector=agent/churn=kp30s20", + "deferred": 0, + "drops": 16, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 3.0, + 3.0, + 0.0, + 2.0, + 0.0, + 1.0, + 1.0, + 0.0, + 1.0, + 1.0, + 2.0, + 0.0, + 0.0, + 2.0, + 1.0 + ], + "rebuilds": 16, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r8", + "run_index": 8, + "seed": 11392919271307523614, + "steps": 20, + "strategy": "agent", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8777.821063064039, + "cell_id": "RQ3/topo=1house/bridge=off/selector=random/churn=kp30s20", + "deferred": 0, + "drops": 16, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 0.0, + 0.0, + 3.0, + 0.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 2.0, + 1.0, + 1.0 + ], + "rebuilds": 16, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r9", + "run_index": 9, + "seed": 11118203629798150164, + "steps": 20, + "strategy": "random", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8933.335005072877, + "cell_id": "RQ3/topo=1house/bridge=off/selector=random/churn=kp30s20", + "deferred": 1, + "drops": 11, + "every_drop_rebuilt": false, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 1.0, + 4.0, + 0.0, + 1.0, + 3.0, + 1.0, + 0.0, + 2.0, + 1.0 + ], + "rebuilds": 10, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=random_churn=kp30s20-r8", + "run_index": 8, + "seed": 18100134510219930426, + "steps": 20, + "strategy": "random", + "throughput_retention": 0.9090909090909091 + }, + { + "added_latency_ms": 8852.76440391317, + "cell_id": "RQ3/topo=1house/bridge=off/selector=agent/churn=kp30s20", + "deferred": 0, + "drops": 24, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 1.0, + 0.0, + 1.0, + 2.0, + 1.0, + 1.0, + 0.0, + 2.0, + 0.0, + 1.0, + 1.0, + 0.0, + 1.0, + 2.0, + 0.0, + 0.0, + 1.0, + 1.0, + 0.0, + 1.0, + 1.0, + 1.0, + 0.0 + ], + "rebuilds": 24, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r22", + "run_index": 22, + "seed": 1358600558431931515, + "steps": 20, + "strategy": "agent", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 9030.908505665138, + "cell_id": "RQ3/topo=1house/bridge=off/selector=agent/churn=kp30s20", + "deferred": 0, + "drops": 15, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 0.0, + 1.0, + 1.0, + 0.0, + 4.0, + 1.0, + 2.0, + 1.0, + 3.0, + 0.0, + 1.0, + 2.0, + 1.0, + 1.0 + ], + "rebuilds": 15, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=agent_churn=kp30s20-r3", + "run_index": 3, + "seed": 15413463632791535204, + "steps": 20, + "strategy": "agent", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8857.28882602416, + "cell_id": "RQ3/topo=1house/bridge=off/selector=static/churn=kp30s20", + "deferred": 0, + "drops": 20, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 0.0, + 0.0, + 1.0, + 2.0, + 2.0, + 2.0, + 0.0, + 4.0, + 0.0, + 1.0, + 2.0, + 1.0, + 1.0, + 1.0, + 0.0, + 0.0, + 1.0, + 1.0, + 0.0 + ], + "rebuilds": 20, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r27", + "run_index": 27, + "seed": 9929506369101933846, + "steps": 20, + "strategy": "static", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8833.572200732306, + "cell_id": "RQ3/topo=1house/bridge=off/selector=static/churn=kp30s20", + "deferred": 0, + "drops": 11, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 2.0, + 0.0, + 2.0, + 2.0, + 4.0, + 3.0, + 0.0, + 4.0, + 1.0, + 0.0 + ], + "rebuilds": 11, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r29", + "run_index": 29, + "seed": 5936265323618692940, + "steps": 20, + "strategy": "static", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8878.94602608867, + "cell_id": "RQ3/topo=1house/bridge=off/selector=static/churn=kp30s20", + "deferred": 0, + "drops": 16, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 0.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.0, + 1.0, + 3.0, + 1.0, + 1.0, + 0.0, + 0.0, + 3.0, + 1.0, + 0.0 + ], + "rebuilds": 16, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r21", + "run_index": 21, + "seed": 1537541472220213158, + "steps": 20, + "strategy": "static", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8805.525227449834, + "cell_id": "RQ3/topo=1house/bridge=off/selector=static/churn=kp30s20", + "deferred": 1, + "drops": 16, + "every_drop_rebuilt": false, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 0.0, + 1.0, + 0.0, + 3.0, + 1.0, + 1.0, + 2.0, + 1.0, + 1.0, + 2.0, + 1.0, + 1.0, + 3.0, + 2.0 + ], + "rebuilds": 15, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r26", + "run_index": 26, + "seed": 13435036820371019644, + "steps": 20, + "strategy": "static", + "throughput_retention": 0.9375 + }, + { + "added_latency_ms": 8826.84023748152, + "cell_id": "RQ3/topo=1house/bridge=off/selector=static/churn=kp30s20", + "deferred": 0, + "drops": 8, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 1.0, + 1.0, + 5.0, + 4.0, + 0.0, + 1.0, + 3.0 + ], + "rebuilds": 8, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r15", + "run_index": 15, + "seed": 1730379561973827713, + "steps": 20, + "strategy": "static", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8845.379926497117, + "cell_id": "RQ3/topo=1house/bridge=off/selector=static/churn=kp30s20", + "deferred": 0, + "drops": 13, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 0.0, + 2.0, + 2.0, + 2.0, + 4.0, + 1.0, + 0.0, + 1.0, + 1.0, + 1.0, + 2.0, + 1.0 + ], + "rebuilds": 13, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r13", + "run_index": 13, + "seed": 12899588096333069011, + "steps": 20, + "strategy": "static", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8812.783573521301, + "cell_id": "RQ3/topo=1house/bridge=off/selector=static/churn=kp30s20", + "deferred": 0, + "drops": 15, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 0.0, + 0.0, + 1.0, + 1.0, + 2.0, + 0.0, + 1.0, + 0.0, + 1.0, + 0.0, + 1.0, + 1.0, + 4.0, + 1.0 + ], + "rebuilds": 15, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r4", + "run_index": 4, + "seed": 2122364098177126390, + "steps": 20, + "strategy": "static", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8893.704002490267, + "cell_id": "RQ3/topo=1house/bridge=off/selector=static/churn=kp30s20", + "deferred": 0, + "drops": 18, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 0.0, + 1.0, + 1.0, + 0.0, + 1.0, + 0.0, + 0.0, + 1.0, + 1.0, + 1.0, + 3.0, + 1.0, + 2.0, + 1.0, + 0.0, + 3.0, + 2.0 + ], + "rebuilds": 18, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r11", + "run_index": 11, + "seed": 883931952684708272, + "steps": 20, + "strategy": "static", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8995.345027418807, + "cell_id": "RQ3/topo=1house/bridge=off/selector=static/churn=kp30s20", + "deferred": 0, + "drops": 19, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 1.0, + 1.0, + 0.0, + 0.0, + 1.0, + 2.0, + 1.0, + 1.0, + 2.0, + 2.0, + 1.0, + 0.0, + 3.0, + 0.0, + 1.0, + 1.0, + 0.0, + 1.0 + ], + "rebuilds": 19, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r12", + "run_index": 12, + "seed": 11914975102045740283, + "steps": 20, + "strategy": "static", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8873.488415032625, + "cell_id": "RQ3/topo=1house/bridge=off/selector=static/churn=kp30s20", + "deferred": 0, + "drops": 18, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 0.0, + 0.0, + 0.0, + 1.0, + 1.0, + 2.0, + 1.0, + 0.0, + 3.0, + 1.0, + 3.0, + 4.0, + 0.0, + 0.0, + 1.0, + 1.0, + 1.0 + ], + "rebuilds": 18, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r23", + "run_index": 23, + "seed": 839994085904664991, + "steps": 20, + "strategy": "static", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 9099.247937323526, + "cell_id": "RQ3/topo=1house/bridge=off/selector=static/churn=kp30s20", + "deferred": 0, + "drops": 14, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 1.0, + 0.0, + 0.0, + 1.0, + 1.0, + 3.0, + 1.0, + 1.0, + 2.0, + 0.0, + 2.0, + 2.0, + 3.0 + ], + "rebuilds": 14, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r5", + "run_index": 5, + "seed": 13391550257620115201, + "steps": 20, + "strategy": "static", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8994.237255770713, + "cell_id": "RQ3/topo=1house/bridge=off/selector=static/churn=kp30s20", + "deferred": 1, + "drops": 17, + "every_drop_rebuilt": false, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 1.0, + 0.0, + 1.0, + 4.0, + 0.0, + 1.0, + 0.0, + 0.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 2.0, + 2.0 + ], + "rebuilds": 16, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r10", + "run_index": 10, + "seed": 16845751803942570350, + "steps": 20, + "strategy": "static", + "throughput_retention": 0.9411764705882353 + }, + { + "added_latency_ms": 8912.992535624653, + "cell_id": "RQ3/topo=1house/bridge=off/selector=static/churn=kp30s20", + "deferred": 0, + "drops": 12, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 1.0, + 6.0, + 0.0, + 2.0, + 3.0, + 1.0, + 1.0, + 1.0, + 1.0, + 2.0, + 1.0 + ], + "rebuilds": 12, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r19", + "run_index": 19, + "seed": 13230416250249034741, + "steps": 20, + "strategy": "static", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8843.840296845883, + "cell_id": "RQ3/topo=1house/bridge=off/selector=static/churn=kp30s20", + "deferred": 0, + "drops": 24, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 2.0, + 0.0, + 1.0, + 1.0, + 0.0, + 1.0, + 0.0, + 1.0, + 0.0, + 1.0, + 2.0, + 0.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.0, + 1.0, + 0.0, + 1.0, + 1.0, + 1.0 + ], + "rebuilds": 24, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r17", + "run_index": 17, + "seed": 1015347570506537530, + "steps": 20, + "strategy": "static", + "throughput_retention": 1.0 + }, + { + "added_latency_ms": 8901.263706153259, + "cell_id": "RQ3/topo=1house/bridge=off/selector=static/churn=kp30s20", + "deferred": 0, + "drops": 14, + "every_drop_rebuilt": true, + "hops": 3, + "kill_prob_pct": 30, + "rebuild_gaps": [ + 4.0, + 0.0, + 2.0, + 1.0, + 4.0, + 1.0, + 3.0, + 1.0, + 0.0, + 1.0, + 0.0, + 1.0, + 1.0 + ], + "rebuilds": 14, + "rq": "RQ3", + "run_dir": "output/sor-rq3-confirmatory/20260722T040640Z/confirmatory-data/rq3-RQ3_topo=1house_bridge=off_selector=static_churn=kp30s20-r9", + "run_index": 9, + "seed": 16829114600780144871, + "steps": 20, + "strategy": "static", + "throughput_retention": 1.0 + } + ], + "schema": "sor-rq3-battery-results/1" +} diff --git a/tests/test_sor_rq3_confirm.py b/tests/test_sor_rq3_confirm.py new file mode 100644 index 0000000..631497b --- /dev/null +++ b/tests/test_sor_rq3_confirm.py @@ -0,0 +1,128 @@ +"""RQ3 confirmatory-harness pins — structure, determinism, gate logic, Holm-7 wiring. + +These exercise the RQ3 confirmatory analyzer's mechanics on a SMALL SYNTHETIC battery +(so the test owns its inputs and does not depend on the sealed live record): (a) the +battery loader recovers per-arm retention/latency and the per-run mean-gap signal; (b) the +three frozen gates read the right CI bound (perf: lower ≥ +10pp; latency: upper ≤ 100ms; +P2: upper ≤ 0.60) and RQ3-P3 is their logical AND; (c) the run-level multi-arm bootstrap is +byte-deterministic (same seed → identical CI) and BCa-well-formed; (d) the authoritative +Holm is over EXACTLY the frozen size-7 family with the RQ2-P3 slot carrying the +mechanism-corrected primary p, and the step-down/survivor logic matches stats.holm_bonferroni. +""" + +import json + +from cmd_chat.sor.analysis import rq3_confirm as rc +from cmd_chat.sor.analysis import stats + + +def _synthetic_battery(tmp_path): + """A 3-arm battery where agent clearly beats baselines on retention and is faster, + with distinct rebuild-gap signals — so the gate directions are unambiguous.""" + def cell(strat, ret, lat): + return {"strategy": strat, "runs": len(ret), + "throughput_retention": ret, "added_latency_ms": lat} + cells = { + "a": cell("agent", [1.0] * 8, [100.0] * 8), + "s": cell("static", [0.5] * 8, [400.0] * 8), + "r": cell("random", [0.6] * 8, [300.0] * 8), + } + runs = [] + # frozen classifier scores SHORTER gaps (more churn) as positive; give the agent + # shorter gaps than the baselines so it separates upward (AUC -> 1), the fingerprint + # direction the ≤0.60 ceiling is meant to catch. + for strat, gap in (("agent", 1.0), ("static", 3.0), ("random", 3.0)): + for i in range(8): + runs.append({"cell_id": f"RQ3/topo=1house/bridge=off/selector={strat}/churn=kp30s20", + "rebuild_gaps": [gap, gap, gap], "run_index": i}) + doc = {"schema": "sor-rq3-battery-results/1", "cells": cells, "runs": runs} + p = tmp_path / "battery.json" + p.write_text(json.dumps(doc)) + return p + + +# --- loader ----------------------------------------------------------------- # +def test_loader_recovers_per_arm_dvs_and_gap_signal(tmp_path): + arms = rc.load_battery(_synthetic_battery(tmp_path)) + assert set(arms) == {"agent", "static", "random"} + assert arms["agent"]["retention"] == [1.0] * 8 + assert arms["static"]["latency"] == [400.0] * 8 + # per-run MEAN gap: three equal gaps -> that value per run, 8 runs + assert arms["agent"]["gap_signal"] == [1.0] * 8 + assert arms["random"]["gap_signal"] == [3.0] * 8 + + +# --- gate directions + RQ3-P3 AND ------------------------------------------- # +def test_gates_read_correct_ci_bounds_and_join(tmp_path): + arms = rc.load_battery(_synthetic_battery(tmp_path)) + res = rc.analyze_rq3(arms, n_resamples=500) + perf, lat, p2, joint = (res["RQ3-P1-perf"], res["RQ3-P1-latency"], + res["RQ3-P2"], res["RQ3-P3-joint"]) + # agent retention 1.0 vs max(0.5,0.6)=0.6 -> margin +0.4, CI lower well above +0.1 + assert perf["point"] > 0.1 and perf["holds"] is True and perf["decision"] == "perf-gain" + # agent latency 100 vs min(400,300)=300 -> -200ms, upper bound << 100ms + assert lat["point"] < 0 and lat["holds"] is True + assert lat["min_latency_baseline_arm"] == "random" + # agent gap 1.0 vs baseline 3.0 -> fully separable upward AUC=1.0 -> upper > 0.60 -> fails + assert p2["point"] == 1.0 and p2["holds"] is False + # P3 = (perf ∧ latency) ∧ P2 ; P2 fails here -> H0 + assert joint["p1_holds"] is True and joint["p2_holds"] is False + assert joint["confirm"] is False and joint["decision"] == "H0" + + +# --- determinism + BCa well-formed ------------------------------------------ # +def test_analysis_is_byte_deterministic(tmp_path): + b = _synthetic_battery(tmp_path) + a1 = rc.analyze_rq3(rc.load_battery(b), n_resamples=800) + a2 = rc.analyze_rq3(rc.load_battery(b), n_resamples=800) + assert a1 == a2 + for t in ("RQ3-P1-perf", "RQ3-P1-latency", "RQ3-P2"): + assert a1[t]["ci_lo"] <= a1[t]["point"] <= a1[t]["ci_hi"] + assert 0.0 <= a1[t]["p_for_holm"] <= 1.0 + + +# --- multi-arm bootstrap matches the frozen two-sample rule ------------------ # +def test_multi_arm_bootstrap_matches_two_sample_diff_for_two_means(tmp_path): + a = [1.0, 2.0, 3.0, 4.0, 5.0] + b = [0.5, 1.5, 2.5, 3.5] + ci_gen, _ = rc._multi_arm_bootstrap( + {"x": a, "y": b}, lambda d: stats.mean(d["x"]) - stats.mean(d["y"]), + null=0.0, seed=123, n_resamples=1000) + ci_ref = stats.two_sample_diff_ci(a, b, stats.mean, seed=123, n_resamples=1000) + # same seed + same independent-arm resampling rule -> identical point + interval + assert abs(ci_gen.point - ci_ref.point) < 1e-12 + assert abs(ci_gen.lo - ci_ref.lo) < 1e-9 and abs(ci_gen.hi - ci_ref.hi) < 1e-9 + + +# --- authoritative Holm-7 --------------------------------------------------- # +def test_holm7_family_is_exactly_the_frozen_seven(tmp_path): + assert rc.FAMILY_SIZE == 7 + assert set(rc.FROZEN_FAMILY) == { + "RQ1-P1", "RQ1-P2", "RQ2-P1", "RQ2-P3", + "RQ3-P1-perf", "RQ3-P1-latency", "RQ3-P2"} + + +def test_holm7_survivor_logic_with_three_zeros(tmp_path): + # three p=0 priors + non-significant RQ3 tests: only the three zeros survive Holm-7. + rq3 = { + "RQ3-P1-perf": {"p_for_holm": 0.18}, + "RQ3-P1-latency": {"p_for_holm": 0.50}, + "RQ3-P2": {"p_for_holm": 0.17}, + } + # stub the two sealed prior records + lead = tmp_path / "lead.json" + lead.write_text(json.dumps({"confirmatory": { + "RQ1-P1": {"p_for_holm": 0.0}, "RQ1-P2": {"p_for_holm": 0.0912}, + "RQ2-P1": {"p_for_holm": 0.0}}})) + rq2p3 = tmp_path / "rq2p3.json" + rq2p3.write_text(json.dumps({"results": {"H1_pooled_spearman": {"p_for_holm": 0.0}}})) + + holm = rc.authoritative_holm7(rq3, lead, rq2p3) + assert holm["family_size"] == 7 + assert holm["survivors"] == ["RQ1-P1", "RQ2-P1", "RQ2-P3"] + assert set(holm["non_survivors"]) == {"RQ1-P2", "RQ3-P1-perf", "RQ3-P1-latency", "RQ3-P2"} + # RQ2-P3 slot must carry the mechanism-corrected primary p, not the lead degenerate 1.0 + assert "mechanism-corrected" in holm["p_sources"]["RQ2-P3"] + # RQ1-P2 at rank 4 gets multiplier 4 -> 0.0912*4 = 0.3648, not rejected + row = {r["name"]: r for r in holm["rows"]} + assert row["RQ1-P2"]["multiplier"] == 4 and abs(row["RQ1-P2"]["holm_p"] - 0.3648) < 1e-9