# Stage-05 analysis clarification — Holm–Bonferroni over the frozen family of 7

**Status: RATIFIED by operator (Andre), 2026-07-19. Worktree clarification note
only — NOT a deviation.** This restates, for the harness, a decision the frozen
prereg **already fixes**; it does **not** edit the frozen prereg
(`sor-consent-prereg.md`, SHA `f22331a72e0d0ccf38b787e63acabbe9d666456ec76076787a6d545c3193425b`).

## Authority (this IS the freeze, not a new choice)

The size-7 family and its disclosure in the lead paper are **pre-registered text**,
so holding `m = 7` for the reported RQ1/RQ2 subset is compliance, not deviation:

- **Prereg §6 — Multiple-comparison correction [APPROVAL]** (prereg lines ~218–222):
  *"the confirmatory hypothesis tests across all three RQ families are corrected
  together by Holm–Bonferroni … Test family (7 confirmatory tests): {RQ1-P1, RQ1-P2,
  RQ2-P1, RQ2-P3, RQ3-P1-perf, RQ3-P1-latency, RQ3-P2}."* The family is frozen at 7.
- **Prereg D6** (prereg line ~317): *"the shared prereg + Holm-corrected family are
  disclosed in whichever papers result (which is more rigorous)."* The lead paper
  discloses the **same size-7 family** even while reporting only the RQ1/RQ2 subset.

Correcting the reported 4 against `m = 4` would shrink the frozen family after the
fact — p-hacking the multiplicity down, contrary to §6. Holding `m = 7` is the
only reading consistent with the frozen text.

## The question

The prereg §6 pre-registers a **confirmatory family of 7** hypothesis tests:

```
RQ1-P1, RQ1-P2, RQ2-P1, RQ2-P3, RQ3-P1-perf, RQ3-P1-latency, RQ3-P2
```

The **lead paper (G4 + RQ1 + RQ2, prereg D6)** reports only the first **4**
(RQ1/RQ2); RQ3's three tests are the severable follow-on (§8). So: when we
Holm-correct the 4 reported p-values, do we correct against **m = 7** (the frozen
family) or **m = 4** (the reported subset)?

Correcting against m = 4 would be **p-hacking the multiplicity down**: the family
was frozen at 7 *before* data, and shrinking it after the fact — because RQ3 is
being reported later — inflates the reported significance of the RQ1/RQ2 subset.
That breaks the pre-registration.

## Decision (implemented default)

**Hold the family size at m = 7 for every reported test.** The reported subset
never shrinks the correction to itself.

Concretely, Holm step-down (`stats.holm_bonferroni`, `confirm.apply_holm`):

- All reported p-values are sorted ascending and assigned ranks `k = 1..R`
  (`R = 4` reported here).
- Each gets multiplier `m − k + 1` with **`m = FROZEN_FAMILY_SIZE = 7`**.
- So the 4 reported tests receive multipliers **7, 6, 5, 4** (smallest p → ×7).
  The step-down enforces monotone non-decreasing adjusted p-values.
- `holm_bonferroni` **raises** if `family_size < #reported` — you can never
  accidentally correct against fewer than the tests you are reporting.

This is a **conservative embedding**: the 4 reported tests are treated as
occupying the 4 *smallest-multiplier* slots (7,6,5,4) of the size-7 family rather
than the true unknown slots they would occupy if all 7 p-values were in hand. The
adjusted p-values are therefore **≥** the values a full simultaneous size-7 Holm
run would assign to these same tests. We over-correct, never under-correct — the
honest direction for a subset report.

*Covered by `tests/test_sor_confirm.py::test_apply_holm_corrects_against_family_of_seven`
(top multiplier == 7; multipliers == [4,5,6,7]) and
`tests/test_sor_stats.py` (family-size monotonicity + `ValueError` when
`family_size < reported`).*

## Gate vs. Holm ordering (not conflated)

Every **reported decision** in §6 is a **CI gate** (RQ1-P1 CI excludes 0.5;
RQ1-P2 ΔAUC CI > 0; RQ2-P1 ΔH two-sided CI sign; RQ2-P3 Spearman ρ CI). The
bootstrap p-values feed the Holm step-down **only to order it** — no reported
claim rests on a bare p. Holm is the family-wise multiplicity guard layered on
top of the pre-registered CI gates, exactly as §6 / rigor-standards §Statistics
require.

## Operator ruling (2026-07-19)

The operator (Andre) ratified the **conservative embedding**: hold `family_size = 7`,
report the 4 RQ1/RQ2 tests, multipliers **7, 6, 5, 4**. This is exactly the frozen
§6 / D6 text above — the harness default now matches the pre-registration.

Two alternatives were considered and **explicitly declined**:

- *Fold RQ3's three cells back in and report all 7 simultaneously.* Statistically
  clean, but RQ3 was dropped for the lead paper (prereg D6) and would need the
  paid/agent selector-arm decisions. **Declined:** do not fold RQ3 back.
- *Full α-splitting / hierarchical gatekeeping.* More powerful, but requires a
  pre-specified α-allocation the frozen prereg does not contain. **Declined:** not
  available without a prereg change (which the freeze forbids).

No open human action remains: the implemented default is the ratified one.