Cross-fit the hit-calling covariance instead of splitting the controls in half
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8
- Forks
- 0
- Avg merge
- 12h 18m
- Merged PRs (30d)
- 26
Description
Problem
hit_calling splits the reference rows in half: one half fits the centroid and covariance, the other supplies the
null pool. That costs power twice over — the covariance comes from half the controls, and the null pool is half
as large as it could be — and it is why _hits.py warns that the covariance is singular at roughly twice as many
features as it otherwise would be.
Cross-fitting keeps every control in the estimate while keeping each scored row out of its own fit: for each fold,
fit on the controls outside it and score the ones inside.
Measured
Prototype against #62's half-split, both with #62's pooled permutation null. Real DMSO cells from jump_cells
relabelled as pseudo-treatments, 96 controls, groups of 24, 10 folds, 60 seeds:
| features | scheme | type-I | power at 0.5 SD | rows in the fit |
|---|---|---|---|---|
| 20 | half-split | 0.031 | 0.731 | 48 |
| 20 | cross-fit | 0.025 | 0.711 | 86 |
| 40 | half-split | 0.047 | 0.892 | 48 |
| 40 | cross-fit | 0.025 | 0.917 | 86 |
| 60 | half-split | 0.056 | 0.850 | 48 |
| 60 | cross-fit | 0.011 | 0.936 | 86 |
Two things follow.
It is a no-op in the normal regime. At 20 features against 48 fit rows the covariance is already comfortable,
and cross-fitting matches the half-split on calibration while losing a little power. Not worth the complexity on
its own.
It pays off where the covariance is starved. As features approach the fit size the half-split loses power
(0.892 to 0.850 from 40 to 60 features) while cross-fitting gains it (0.917 to 0.936). That is the regularization
ceasing to dominate the distances, and it is the same regime _hits.py already warns about.
The blocker
Cross-fitting drifts conservative as features grow — 0.011 at 60 features, five times below nominal. So it trades
the half-split's anti-conservative drift for a conservative one rather than fixing calibration.
The likely cause is in the prototype rather than the method: control rows were scored by a fit excluding their own
fold, treated rows by a fit on all controls. Those two are not quite exchangeable, and the gap widens as the
covariance gets harder to estimate. Resolving that is the real work here — score both sides under the same fits,
or fold the treated rows into the scheme — and it has to be settled before this can ship, since the whole point of
#62 was a null that is exchangeable with what it is compared against.
Proposed fix
- Score treated and control rows under the same fits so the two sides stay exchangeable
- Re-measure calibration and power across features from 10 to 60 against the half-split
- Decide the fold count, or use leave-one-out with a rank-1 downdate of the covariance
- Only adopt it if calibration holds across the range; a conservative test is not an improvement on an
anti-conservative one, only a different error - If adopted, revisit the singular-covariance warning, whose threshold is written for the half-split
Until then the guidance in #62's docstring stands: reduce the features with pp.feature_select or score
use_rep="X_pca", which puts a screen in the regime where both schemes agree.
Follows from #60 and #62.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the hit-calling implementation in _hits.py and the guidance in #62's docstring. Reproduce the reported cross-fitting measurements, then compare schemes that score treated and control rows under the same fits. Done means a settled fold or downdate design with calibration and power measurements across 10–60 features, and adoption only if calibration holds.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100