Benjamini-Hochberg Adjustment across two Families #18

New Issue

GKaszewski · 2026-08-26T09:15:49Z

GKaszewski commented

2026-08-26 09:15:49 +00:00

What to build

An Adjustment layer that corrects CorrelationStrategy results for how many things were tested at once, and marks results rather than hiding them.

The problem is arithmetic, not a flaw in the data. A realistic stats page tests roughly 8 metrics plus cycle day, weather and moon, plus however many Activities the User keeps — call it 41. At a conventional 5% threshold, the chance of at least one false positive when nothing real is happening is 1 - 0.95^41, about 88%.

Benjamini-Hochberg, controlling false discovery rate — not family-wise error control. A false positive costs a User a behaviour change that achieves nothing; a false negative costs them a real and actionable effect they never discover. In a personal exploratory tool the second is worse. Strict family-wise control needs roughly |r| >= 0.34 over ninety days, which would leave the page empty for months and read as broken rather than rigorous.

Nothing is gated. A surviving result is presented as a second axis of robustness beside cross-strategy agreement — it held up once the number of things tested was accounted for — never as a verdict or a significance claim. This keeps ADR 0006's contract intact.

Two rules that follow from the maths rather than from preference:

Within a strategy, never across. Pearson, Spearman and Kendall are three measurements of one hypothesis, not three hypotheses. Correcting across them would penalise measuring carefully.
MetricKinds and Activities are separate Familys. A Family is a question, and how many Activities a User keeps has no bearing on whether their sleep tracks their mood. A single Family would let a large Activity catalog quietly suppress metric findings, making the guardrail's strictness depend on a number the User controls for unrelated reasons.

Acceptance criteria

Benjamini-Hochberg is implemented over a set of results
Adjustment marks results; nothing is ever hidden or filtered out by it
Correction runs within a single CorrelationStrategy, never across strategies
MetricKinds and Activities are corrected as separate Familys
Adding Activities does not change whether a metric result is marked
The q threshold is configuration, not a constant
Adjustment is not reachable as a CorrelationStrategy
The mark reads as robustness, not as a significance claim

Blocked by

Remaining CorrelationStrategies and agreement display (#17)

## What to build An `Adjustment` layer that corrects `CorrelationStrategy` results for how many things were tested at once, and **marks** results rather than hiding them. The problem is arithmetic, not a flaw in the data. A realistic stats page tests roughly 8 metrics plus cycle day, weather and moon, plus however many Activities the User keeps — call it 41. At a conventional 5% threshold, the chance of at least one false positive when nothing real is happening is `1 - 0.95^41`, about 88%. **Benjamini-Hochberg, controlling false discovery rate** — not family-wise error control. A false positive costs a User a behaviour change that achieves nothing; a false negative costs them a real and actionable effect they never discover. In a personal exploratory tool the second is worse. Strict family-wise control needs roughly |r| >= 0.34 over ninety days, which would leave the page empty for months and read as broken rather than rigorous. **Nothing is gated.** A surviving result is presented as a second axis of robustness beside cross-strategy agreement — it held up once the number of things tested was accounted for — never as a verdict or a significance claim. This keeps ADR 0006's contract intact. Two rules that follow from the maths rather than from preference: - **Within a strategy, never across.** Pearson, Spearman and Kendall are three measurements of one hypothesis, not three hypotheses. Correcting across them would penalise measuring carefully. - **`MetricKind`s and Activities are separate `Family`s.** A `Family` is a question, and how many Activities a User keeps has no bearing on whether their sleep tracks their mood. A single `Family` would let a large Activity catalog quietly suppress metric findings, making the guardrail's strictness depend on a number the User controls for unrelated reasons. ## Acceptance criteria - [ ] Benjamini-Hochberg is implemented over a set of results - [ ] `Adjustment` marks results; nothing is ever hidden or filtered out by it - [ ] Correction runs within a single `CorrelationStrategy`, never across strategies - [ ] `MetricKind`s and Activities are corrected as separate `Family`s - [ ] Adding Activities does not change whether a metric result is marked - [ ] The q threshold is configuration, not a constant - [ ] `Adjustment` is not reachable as a `CorrelationStrategy` - [ ] The mark reads as robustness, not as a significance claim ## Blocked by - #17

GKaszewski added the ready-for-agent label 2026-08-26 09:15:49 +00:00

GKaszewski closed this issue

2026-08-26 19:07:01 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: GKaszewski/k-mood#18