hack-house/docs/rq3-companion-run-brief.md

Execution-freeze / run-brief (DRAFT) — RQ3 companion battery

STATUS: PARAMS LOCKED 2026-07-21 — execution plan, NOT a new prereg. RQ3's hypotheses, gates, and analysis are already frozen in sor-consent-prereg.md (SHA f22331a72e0d0ccf38b787e63acabbe9d666456ec76076787a6d545c3193425b, §3/§4/§6). Writing a "new RQ3 prereg" would be redundant and risk HARKing. This brief (a) pins the two open [APPROVAL] execution parameters (agent model, churn schedule — both locked below, blind to RQ3 outcomes) and (b) lists the build + gate steps. GO gate before the confirmatory battery: the rebuild-classifier calibration gate + the churn-bites gate must pass green (§3-4). Build + synthetic tests MAY proceed now.

1. The frozen gates (verbatim — do not restate as new)

Family stays the frozen size-7 {RQ1-P1, RQ1-P2, RQ2-P1, RQ2-P3, RQ3-P1-perf, RQ3-P1-latency, RQ3-P2}. The three RQ3 tests were always in the family.

Test	Frozen gate
RQ3-P1-perf	throughput-retention(agent) − max(static,random): CI lower bound ≥ 10 pp
RQ3-P1-latency	added-latency(agent): CI upper bound ≤ 100 ms
RQ3-P2	rebuild-classifier AUC: CI upper bound ≤ 0.60
RQ3-P3	logical AND: CONFIRM iff P1 ∧ P2 (not a new test)

R = 30 runs/cell, C = 50 circuits/run, fixed stopping rule (no optional stopping / interim looks; uninformative cell → inconclusive). All inherited unchanged.

2. Open [APPROVAL] params to PIN before GO (the actual gate)

The frozen prereg left two items tagged [APPROVAL] pinned at freeze — the [APPROVAL] tag is still literally present in the frozen text, i.e. neither was concretely substituted. These must be pinned now, blind to RQ3 data:

(A) Agent selector model ID + weights digest (§3 L71-73) — LOCKED 2026-07-21.

• Pin: qwen2.5:3b via local Ollama (agent_selector.OllamaAgentPolicy, temp 0, per-run seed, state-hash cache; heuristic fallback on query failure). Record the exact model tag + SHA-256 weights digest in every manifest (digest captured at run time).
• Rails satisfied: local / open-weight, $0, no paid-frontier call. The Claude/frontier arm (ClaudeExploratoryPolicy) stays inert / EXPLORATORY / budget-gated — not wired.
• Reproducibility caveat (accepted; must be stated in the paper). Ollama at temp 0 is not bit-identical across machines (quantization / GPU logit drift). The agent arm is reproducible via the committed decision-log + state-hash cache replay, not via independent model re-execution on other hardware — same honesty class as the RQ1 timing caveat. Commit the decision log + cache as the reproducibility anchor.

(B) Churn schedule set (§3 L74, §4) — LOCKED 2026-07-21. churn.churn_schedule(seed, nodes, steps, kill_prob_pct) is implemented and deterministic.

• Pin: a seeded set of schedules — one schedule per run, seed drawn from the same SHA256(S0‖cell‖run) family so every run's churn is reproducible and provenance-anchored. kill_prob_pct = 30, steps = 20.
• Churn must actually bite (gate, §4). With these values, confirm on a dry pass that circuits genuinely lose hops and rebuild (non-zero drops/rebuilds per run). If drop rate is trivially zero, the throughput-retention and rebuild-classifier tests are degenerate — STOP and surface rather than report a zero-variance null.

3. Build gaps (hand to the driver in tmux; not built inline)

Verified against the current worktree — what exists vs. what's missing:

Ready (no work): selector.py static/random/agent all implemented + tested; agent_selector.OllamaAgentPolicy reproducible; churn.py deterministic schedules wired into run_selection(); analysis/metrics.py has throughput_retention() + rebuild_classifier_auc().

Missing (build):

1. battery.enumerate_cells() (battery.py:73) enumerates RQ1+RQ2 only (6 cells). Add RQ3 cells: selector ∈ {static, random, agent} × churn-schedule(s) at the 1house/bridge-off control; interleave the control arm like RQ1/RQ2.
2. executor.run_battery() — wire RQ3 collection: drive run_selection() over the pinned schedule; collect throughput-retention, added-latency, and rebuild-event gaps → classifier AUC. Baselines LOCKED: added-latency = median e2e latency(agent) − median latency(min-latency baseline arm, i.e. the faster of static/random); perf margin uses max(static, random) per the frozen gate.
3. Rebuild-classifier calibration gate — calibrate the classifier on labeled churned-vs-low-churn control signals before the confirmatory RQ3 battery (same discipline as the RQ1 correlator gate). Low-churn baseline LOCKED: kill_prob_pct = 5. Gate passes iff churned-vs-baseline is separable (AUC≈1) and baseline-vs-baseline is not (AUC≈0.5). Boolean; blocks the run.

4. Instrument-validation gate (boolean; blocks RQ3 confirmatory)

Reuse the frozen 6-item gate spirit, plus RQ3-specific:

• Rebuild classifier calibrated on control signals (AUC≈1 churned-vs-baseline separable, AUC≈0.5 baseline-vs-baseline) — not fit to confirmatory cells.
• Agent selector reproducibility check: same seed + state-hash → byte-identical circuit choice (cache replay), model+digest echoed to manifest.
• Isolation (assert engine != local), provenance integrity (sealed events.jsonl).

5. Analysis

• Reuse stage06_run on the sealed RQ3 data; compute the three RQ3 tests with effect + BCa CI.
• Complete the Holm step-down over the full family of 7 now that all seven p-values exist. (The lead paper's 7/6/5/4 was a deliberately conservative partial embedding; the companion computes the exact Holm-7 — both remain valid, the partial never under-corrects.) The companion's Holm-7 is the authoritative final correction over the whole family, including the lead RQ1/RQ2 tests; RQ1-P1 and RQ2-P1 survive regardless (raw p ≈ 0), but state explicitly that the companion supersedes the lead's partial embedding.
• Confirmatory vs exploratory labeled; nulls reported honestly (a selector that does not beat baselines, or a rebuild pattern that is a fingerprint, is the finding).

6. QUIC / ssh3 transport arm — stays EXPLORATORY

Frozen as EXPLORATORY, orthogonal to all three RQs, and not built (design decision D3; §3 L75-78). Recommendation: defer. Promoting it doubles every cell + requires a new transport substrate for a weak lit-gap. If ever built, carry only as a labeled EXPLORATORY latency-sensitivity note — never in the Holm family.

7. Rails (immutable, inherited)

Frozen prereg never edited; execution pins recorded here + in manifests; no HARKing; nulls are results. Containment: isolated docker only, self-traffic, lab-only. Budget $0 — local open-weight agent only, no paid arm; surface before any spend. Worktree only on feat/sor-consent-relay. Raw data immutable + SHA-256.

8. Pin block (to complete at GO)

GO DATE: <date>
AGENT MODEL: qwen2.5:3b   WEIGHTS DIGEST: <sha256>
CHURN: kill_prob_pct=30  steps=<S>  schedules=<seeded set spec>
APPROVED BY: <operator>
FROZEN PREREG SHA (unchanged): f22331a72e0d0ccf38b787e63acabbe9d666456ec76076787a6d545c3193425b