bigbio / bigbio/hvantk

rerank: port the multiplicity control, blocked CV and seed handling into the library

Open
#247 0 comments 0 reactions 1 assignee View on GitHub

@ypriverol is already working on this.

Since Aug 7, 2026.

Dominant language
Python
Stars
1
Forks
0
Avg merge
1d 3h
Merged PRs (30d)
40

Description

## Problem

`hvantk rerank` ships the model and one of three controls. A 52-arm study
(`hvantk-research analysis/rerank-homogenised`, 5 cohorts × 4 panels × 4 feature
representations, each with a 200-permutation null) imports exactly **four symbols** from the
library:

```python
from hvantk.algorithms.rerank.evaluator import _raw_oof, _gbm
from hvantk.algorithms.rerank.leakage import leakage_selector, resolve_leakage
```

Everything that decides whether a result is real — the multiplicity correction, the blocked
CV, the seed sweep — lives in analysis scripts outside the repo. A user running
`hvantk rerank` today gets a number with no way to ask whether it survives selection.

### The control stack is 2/3 ported

| control | asks | library | state |
|---|---|---|---|
| Circularity | what was this predictor **trained** on? | `provenance.py` | ✅ |
| Presence-leakage | which genes was it **run** on? | `leakage.py` (#244) | ✅ |
| **Multiplicity** | we searched N axes and reported the best — could that arise by chance? | — | ❌ **absent** |

`grep -rn "permut\|selected_max" hvantk/algorithms/rerank/ hvantk/tools/rerank/` returns
nothing.

This is not a theoretical gap. The selected-maximum null has reversed **four** results in this
project, and the selected-max null median is routinely +0.012 to +0.027 — larger than most
axes' entire measured gain. Without it, "axis X adds +0.02" is not interpretable.

---

## Pass A — permutation nulls and the selected-maximum correction

**New module** `hvantk/algorithms/rerank/nulls.py`.

Two distributions, because they answer different questions:

- **per-axis null** — could *this* axis's delta arise by chance?
- **selected-maximum null** — the distribution of the *largest* delta achieved by **any**
offered axis under permutation. Only this one is a multiplicity correction.

Contract notes that matter and are easy to get wrong:

- Each permutation must **refit the baseline too**. Comparing a permuted-label augmented model
against an unpermuted baseline measures the permutation, not the axis.
- p-values use `(1 + #{null >= obs}) / (1 + n_perm)`. A finite permutation set can never
license `p = 0`.
- The null is generated under **one** control setting. Attaching it to deltas computed under a
different feature set (circularity arm, leakage off) is a category error and should raise,
not silently mismatch.
- Chunkable for parallel execution: `n_perm`, `chunk`, `n_chunks`.

**Acceptance:** a permuted-label dataset yields selected-max p ≈ uniform; an axis with a
planted signal clears; an arm offered more axes gets a higher null median than the same arm
offered fewer.

## Pass B — blocked cross-validation

`StratifiedGroupKFold` + a `groups=` path exist only in analysis code. `evaluator._raw_oof`
uses plain `StratifiedKFold`, so gene families straddle folds and pooled OOF AUC is optimistic
wherever paralogues share a label.

- Thread `groups` through `ReRanker.score`, `_raw_oof` and the ablation path.
- Ship the block builder (HGNC primary `gene_group`) plus a **hard abort** when the largest
block exceeds a configurable fraction of n — a dominant block makes pooled OOF AUC
meaningless, and silently continuing is worse than failing.

**Acceptance:** with paralogue groups supplied, no fold contains two members of one block;
the abort fires above the ceiling.

## Pass C — the seed is a parameter, and the CI covers more than one component

`random_state=42` is hardcoded at `evaluator.py:17`, `reranker.py:32`, `selection.py:177`, and
`_boot_ci` seeds `default_rng(42)`. The shipped interval resamples **genes only** — one of at
least three variance components:

| component | covered today |
|---|---|
| gene resampling | ✅ |
| CV partition (which genes land in which fold) | ❌ one hardcoded seed |
| axis selection (best-of-N) | ❌ Pass A |

Measured consequence: a single-seed run scored 0.765 against a 25-seed mean of **0.784** — the
published figure was the unluckiest draw, and nothing in the API let a user notice.

- `seed` becomes a config field, threaded everywhere; no bare `42`.
- Add a multi-seed evaluation returning the across-seed spread alongside the bootstrap CI.

**Acceptance:** two different seeds give different partitions; the reported interval widens
when the seed sweep is enabled.

---

## Housekeeping

`evaluator.py:1` still reads `# local/rerank_engine/evaluator.py` — the file was moved from a
prototype and never reviewed. `evaluator.py` is 59 lines of dense one-liners with several
statements per line; it is the module these three passes touch most.

## Order

A → B → C. Pass A is the one that changes what conclusions users can draw; B and C change how
wide the error bars are.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.