RQ1+RQ2 start-line: live per-cell condition assembler + guarded launcher + Holm ratification
Wire the live per-cell condition assembler (cmd_chat/sor/assembler.py): compose
the existing R1/R4/R5/R6 pieces into a condition-encoding CircuitSpec per frozen
§2 cell, so a cell_id maps to a genuinely distinct, ForwarderPlan-gated
(engine != local) isolated circuit rather than the plain 3-hop control:
- RQ1 bridge-on / on+padding insert a live bridge hop (+ R1 PADDING stream);
- RQ2 bridge-/directory-federated genuinely span >= 2 houses
(federation.select_federated_path, split-knowledge).
Plans only: opens no socket, moves no traffic, stands up no engine.
battery.assembler_dry_check validates on FIXTURES (6/6 cells distinct
fingerprints, all isolation-gated, same (cell,seed) reproduces, RQ1 bridge +
padding arms live, RQ2 federation >= 2 houses); write-once, kept out of any
confirmatory data dir. confirmatory_run preflight now runs it; the --operator-go
path no longer refuses cells as "not wired" — triple-lock + green preflight HOLD
on grid completion, then surface the operator's immutable data-run gate. No
confirmatory cell is ever fabricated.
Start-line instrument-validation gate (cmd_chat/sor/gate.py), grid + containment
pin (grid.py), and the guarded launcher (confirmatory_run.py) land alongside.
Holm: hold family_size = 7, report the 4 RQ1/RQ2 tests -> multipliers 7,6,5,4.
This is the pre-registration, not a deviation (prereg §6 [APPROVAL] size-7 family
+ D6 disclosure in the lead paper); docs/stage-05-holm-clarification.md marked
RATIFIED. RQ3-fold and alpha-split declined.
Containment intact; prereg untouched (SHA f22331a72e...); worktree-only. output/
gitignored (runtime artifacts, never source). Full SOR suite 156 passed. No
confirmatory data collected — the live data run remains the human gate.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,94 @@
|
||||
# Stage-05 analysis clarification — Holm–Bonferroni over the frozen family of 7
|
||||
|
||||
**Status: RATIFIED by operator (Andre), 2026-07-19. Worktree clarification note
|
||||
only — NOT a deviation.** This restates, for the harness, a decision the frozen
|
||||
prereg **already fixes**; it does **not** edit the frozen prereg
|
||||
(`sor-consent-prereg.md`, SHA `f22331a72e0d0ccf38b787e63acabbe9d666456ec76076787a6d545c3193425b`).
|
||||
|
||||
## Authority (this IS the freeze, not a new choice)
|
||||
|
||||
The size-7 family and its disclosure in the lead paper are **pre-registered text**,
|
||||
so holding `m = 7` for the reported RQ1/RQ2 subset is compliance, not deviation:
|
||||
|
||||
- **Prereg §6 — Multiple-comparison correction [APPROVAL]** (prereg lines ~218–222):
|
||||
*"the confirmatory hypothesis tests across all three RQ families are corrected
|
||||
together by Holm–Bonferroni … Test family (7 confirmatory tests): {RQ1-P1, RQ1-P2,
|
||||
RQ2-P1, RQ2-P3, RQ3-P1-perf, RQ3-P1-latency, RQ3-P2}."* The family is frozen at 7.
|
||||
- **Prereg D6** (prereg line ~317): *"the shared prereg + Holm-corrected family are
|
||||
disclosed in whichever papers result (which is more rigorous)."* The lead paper
|
||||
discloses the **same size-7 family** even while reporting only the RQ1/RQ2 subset.
|
||||
|
||||
Correcting the reported 4 against `m = 4` would shrink the frozen family after the
|
||||
fact — p-hacking the multiplicity down, contrary to §6. Holding `m = 7` is the
|
||||
only reading consistent with the frozen text.
|
||||
|
||||
## The question
|
||||
|
||||
The prereg §6 pre-registers a **confirmatory family of 7** hypothesis tests:
|
||||
|
||||
```
|
||||
RQ1-P1, RQ1-P2, RQ2-P1, RQ2-P3, RQ3-P1-perf, RQ3-P1-latency, RQ3-P2
|
||||
```
|
||||
|
||||
The **lead paper (G4 + RQ1 + RQ2, prereg D6)** reports only the first **4**
|
||||
(RQ1/RQ2); RQ3's three tests are the severable follow-on (§8). So: when we
|
||||
Holm-correct the 4 reported p-values, do we correct against **m = 7** (the frozen
|
||||
family) or **m = 4** (the reported subset)?
|
||||
|
||||
Correcting against m = 4 would be **p-hacking the multiplicity down**: the family
|
||||
was frozen at 7 *before* data, and shrinking it after the fact — because RQ3 is
|
||||
being reported later — inflates the reported significance of the RQ1/RQ2 subset.
|
||||
That breaks the pre-registration.
|
||||
|
||||
## Decision (implemented default)
|
||||
|
||||
**Hold the family size at m = 7 for every reported test.** The reported subset
|
||||
never shrinks the correction to itself.
|
||||
|
||||
Concretely, Holm step-down (`stats.holm_bonferroni`, `confirm.apply_holm`):
|
||||
|
||||
- All reported p-values are sorted ascending and assigned ranks `k = 1..R`
|
||||
(`R = 4` reported here).
|
||||
- Each gets multiplier `m − k + 1` with **`m = FROZEN_FAMILY_SIZE = 7`**.
|
||||
- So the 4 reported tests receive multipliers **7, 6, 5, 4** (smallest p → ×7).
|
||||
The step-down enforces monotone non-decreasing adjusted p-values.
|
||||
- `holm_bonferroni` **raises** if `family_size < #reported` — you can never
|
||||
accidentally correct against fewer than the tests you are reporting.
|
||||
|
||||
This is a **conservative embedding**: the 4 reported tests are treated as
|
||||
occupying the 4 *smallest-multiplier* slots (7,6,5,4) of the size-7 family rather
|
||||
than the true unknown slots they would occupy if all 7 p-values were in hand. The
|
||||
adjusted p-values are therefore **≥** the values a full simultaneous size-7 Holm
|
||||
run would assign to these same tests. We over-correct, never under-correct — the
|
||||
honest direction for a subset report.
|
||||
|
||||
*Covered by `tests/test_sor_confirm.py::test_apply_holm_corrects_against_family_of_seven`
|
||||
(top multiplier == 7; multipliers == [4,5,6,7]) and
|
||||
`tests/test_sor_stats.py` (family-size monotonicity + `ValueError` when
|
||||
`family_size < reported`).*
|
||||
|
||||
## Gate vs. Holm ordering (not conflated)
|
||||
|
||||
Every **reported decision** in §6 is a **CI gate** (RQ1-P1 CI excludes 0.5;
|
||||
RQ1-P2 ΔAUC CI > 0; RQ2-P1 ΔH two-sided CI sign; RQ2-P3 Spearman ρ CI). The
|
||||
bootstrap p-values feed the Holm step-down **only to order it** — no reported
|
||||
claim rests on a bare p. Holm is the family-wise multiplicity guard layered on
|
||||
top of the pre-registered CI gates, exactly as §6 / rigor-standards §Statistics
|
||||
require.
|
||||
|
||||
## Operator ruling (2026-07-19)
|
||||
|
||||
The operator (Andre) ratified the **conservative embedding**: hold `family_size = 7`,
|
||||
report the 4 RQ1/RQ2 tests, multipliers **7, 6, 5, 4**. This is exactly the frozen
|
||||
§6 / D6 text above — the harness default now matches the pre-registration.
|
||||
|
||||
Two alternatives were considered and **explicitly declined**:
|
||||
|
||||
- *Fold RQ3's three cells back in and report all 7 simultaneously.* Statistically
|
||||
clean, but RQ3 was dropped for the lead paper (prereg D6) and would need the
|
||||
paid/agent selector-arm decisions. **Declined:** do not fold RQ3 back.
|
||||
- *Full α-splitting / hierarchical gatekeeping.* More powerful, but requires a
|
||||
pre-specified α-allocation the frozen prereg does not contain. **Declined:** not
|
||||
available without a prereg change (which the freeze forbids).
|
||||
|
||||
No open human action remains: the implemented default is the ratified one.
|
||||
Reference in New Issue
Block a user