scverse / scverse/mantispy

Cross-fit the hit-calling covariance instead of splitting the controls in half

Open
#64 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
8
Forks
0
Avg merge
12h 18m
Merged PRs (30d)
26

Description

Problem

hit_calling splits the reference rows in half: one half fits the centroid and covariance, the other supplies the
null pool. That costs power twice over — the covariance comes from half the controls, and the null pool is half
as large as it could be — and it is why _hits.py warns that the covariance is singular at roughly twice as many
features as it otherwise would be.

Cross-fitting keeps every control in the estimate while keeping each scored row out of its own fit: for each fold,
fit on the controls outside it and score the ones inside.

Measured

Prototype against #62's half-split, both with #62's pooled permutation null. Real DMSO cells from jump_cells
relabelled as pseudo-treatments, 96 controls, groups of 24, 10 folds, 60 seeds:

features scheme type-I power at 0.5 SD rows in the fit
20 half-split 0.031 0.731 48
20 cross-fit 0.025 0.711 86
40 half-split 0.047 0.892 48
40 cross-fit 0.025 0.917 86
60 half-split 0.056 0.850 48
60 cross-fit 0.011 0.936 86

Two things follow.

It is a no-op in the normal regime. At 20 features against 48 fit rows the covariance is already comfortable,
and cross-fitting matches the half-split on calibration while losing a little power. Not worth the complexity on
its own.

It pays off where the covariance is starved. As features approach the fit size the half-split loses power
(0.892 to 0.850 from 40 to 60 features) while cross-fitting gains it (0.917 to 0.936). That is the regularization
ceasing to dominate the distances, and it is the same regime _hits.py already warns about.

The blocker

Cross-fitting drifts conservative as features grow — 0.011 at 60 features, five times below nominal. So it trades
the half-split's anti-conservative drift for a conservative one rather than fixing calibration.

The likely cause is in the prototype rather than the method: control rows were scored by a fit excluding their own
fold, treated rows by a fit on all controls. Those two are not quite exchangeable, and the gap widens as the
covariance gets harder to estimate. Resolving that is the real work here — score both sides under the same fits,
or fold the treated rows into the scheme — and it has to be settled before this can ship, since the whole point of
#62 was a null that is exchangeable with what it is compared against.

Proposed fix

  • Score treated and control rows under the same fits so the two sides stay exchangeable
  • Re-measure calibration and power across features from 10 to 60 against the half-split
  • Decide the fold count, or use leave-one-out with a rank-1 downdate of the covariance
  • Only adopt it if calibration holds across the range; a conservative test is not an improvement on an
    anti-conservative one, only a different error
  • If adopted, revisit the singular-covariance warning, whose threshold is written for the half-split

Until then the guidance in #62's docstring stands: reduce the features with pp.feature_select or score
use_rep="X_pca", which puts a screen in the regime where both schemes agree.

Follows from #60 and #62.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the hit-calling implementation in _hits.py and the guidance in #62's docstring. Reproduce the reported cross-fitting measurements, then compare schemes that score treated and control rows under the same fits. Done means a settled fold or downdate design with calibration and power measurements across 10–60 features, and adoption only if calibration holds.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.