Benchmarking & evaluation harness for perturbation prediction (baseline-aware)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 345
- Forks
- 66
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 13
Description
Context
As we add perturbation-prediction models, we need a standardized, honest way to evaluate them.
There is strong evidence in the field that trivial baselines (control mean, additive model) often match or beat elaborate models, so baselines must be first-class, not an afterthought.
PerturbationComparison is a start but is limited.
What's missing
- Standard train/test splits for perturbation prediction: held-out perturbations, held-out combinations, held-out cell types.
- Built-in baselines: control mean, additive/linear, nearest-perturbation (reuse
DistanceSpace.nearest_perturbations). - Standard metrics: per-DEG delta / logFC correlation, E-distance to ground truth, direction (sign) accuracy, top-k DEG overlap.
- A simple leaderboard-style summary over models × metrics × splits.
Proposal / API
A pertpy.tools evaluator that takes ground-truth and predicted AnnData (as returned by the prediction models) and returns a tidy results DataFrame, reusing the existing Distance metrics.
Why it matters
Gives users — and reviewers — an honest, reproducible answer to "is this model actually better than doing nothing clever?".
This is exactly the rigor scverse is trusted for, and it is the natural companion to a broader prediction module.
Related
Prior discussion in #173 (benchmarking tool speed) is about runtime, not predictive accuracy — this is complementary.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing PerturbationComparison and the existing Distance metrics, then inspect DistanceSpace.nearest_perturbations. The proposed entry point is a pertpy.tools evaluator accepting ground-truth and predicted AnnData and returning a tidy results DataFrame. Done means the requested splits, baselines, metrics, and leaderboard-style summary are standardized and covered by appropriate tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100