rerank: port the multiplicity control, blocked CV and seed handling into the library
@ypriverol is already working on this.
Since Aug 7, 2026.
- Dominant language
- Python
- Stars
- 1
- Forks
- 0
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 40
Description
## Problem
`hvantk rerank` ships the model and one of three controls. A 52-arm study
(`hvantk-research analysis/rerank-homogenised`, 5 cohorts × 4 panels × 4 feature
representations, each with a 200-permutation null) imports exactly **four symbols** from the
library:
```python
from hvantk.algorithms.rerank.evaluator import _raw_oof, _gbm
from hvantk.algorithms.rerank.leakage import leakage_selector, resolve_leakage
```
Everything that decides whether a result is real — the multiplicity correction, the blocked
CV, the seed sweep — lives in analysis scripts outside the repo. A user running
`hvantk rerank` today gets a number with no way to ask whether it survives selection.
### The control stack is 2/3 ported
| control | asks | library | state |
|---|---|---|---|
| Circularity | what was this predictor **trained** on? | `provenance.py` | ✅ |
| Presence-leakage | which genes was it **run** on? | `leakage.py` (#244) | ✅ |
| **Multiplicity** | we searched N axes and reported the best — could that arise by chance? | — | ❌ **absent** |
`grep -rn "permut\|selected_max" hvantk/algorithms/rerank/ hvantk/tools/rerank/` returns
nothing.
This is not a theoretical gap. The selected-maximum null has reversed **four** results in this
project, and the selected-max null median is routinely +0.012 to +0.027 — larger than most
axes' entire measured gain. Without it, "axis X adds +0.02" is not interpretable.
---
## Pass A — permutation nulls and the selected-maximum correction
**New module** `hvantk/algorithms/rerank/nulls.py`.
Two distributions, because they answer different questions:
- **per-axis null** — could *this* axis's delta arise by chance?
- **selected-maximum null** — the distribution of the *largest* delta achieved by **any**
offered axis under permutation. Only this one is a multiplicity correction.
Contract notes that matter and are easy to get wrong:
- Each permutation must **refit the baseline too**. Comparing a permuted-label augmented model
against an unpermuted baseline measures the permutation, not the axis.
- p-values use `(1 + #{null >= obs}) / (1 + n_perm)`. A finite permutation set can never
license `p = 0`.
- The null is generated under **one** control setting. Attaching it to deltas computed under a
different feature set (circularity arm, leakage off) is a category error and should raise,
not silently mismatch.
- Chunkable for parallel execution: `n_perm`, `chunk`, `n_chunks`.
**Acceptance:** a permuted-label dataset yields selected-max p ≈ uniform; an axis with a
planted signal clears; an arm offered more axes gets a higher null median than the same arm
offered fewer.
## Pass B — blocked cross-validation
`StratifiedGroupKFold` + a `groups=` path exist only in analysis code. `evaluator._raw_oof`
uses plain `StratifiedKFold`, so gene families straddle folds and pooled OOF AUC is optimistic
wherever paralogues share a label.
- Thread `groups` through `ReRanker.score`, `_raw_oof` and the ablation path.
- Ship the block builder (HGNC primary `gene_group`) plus a **hard abort** when the largest
block exceeds a configurable fraction of n — a dominant block makes pooled OOF AUC
meaningless, and silently continuing is worse than failing.
**Acceptance:** with paralogue groups supplied, no fold contains two members of one block;
the abort fires above the ceiling.
## Pass C — the seed is a parameter, and the CI covers more than one component
`random_state=42` is hardcoded at `evaluator.py:17`, `reranker.py:32`, `selection.py:177`, and
`_boot_ci` seeds `default_rng(42)`. The shipped interval resamples **genes only** — one of at
least three variance components:
| component | covered today |
|---|---|
| gene resampling | ✅ |
| CV partition (which genes land in which fold) | ❌ one hardcoded seed |
| axis selection (best-of-N) | ❌ Pass A |
Measured consequence: a single-seed run scored 0.765 against a 25-seed mean of **0.784** — the
published figure was the unluckiest draw, and nothing in the API let a user notice.
- `seed` becomes a config field, threaded everywhere; no bare `42`.
- Add a multi-seed evaluation returning the across-seed spread alongside the bootstrap CI.
**Acceptance:** two different seeds give different partitions; the reported interval widens
when the seed sweep is enabled.
---
## Housekeeping
`evaluator.py:1` still reads `# local/rerank_engine/evaluator.py` — the file was moved from a
prototype and never reviewed. `evaluator.py` is 59 lines of dense one-liners with several
statements per line; it is the module these three passes touch most.
## Order
A → B → C. Pass A is the one that changes what conclusions users can draw; B and C change how
wide the error bars are.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.