Evaluate AlphaPeptDeep ONNX models (MS2 intensity + RT) as additive rescoring features
- Dominant language
- Rust
- Stars
- 2
- Forks
- 2
- Avg merge
- 10h 11m
- Merged PRs (30d)
- 42
Description
Prompted by OpenMS/OpenMS#9975, which adds AlphaPeptDeep prediction-based rescoring
to ProSE: two pre-trained ONNX models (MS2 fragment intensity, retention time),
executed through ONNX Runtime behind a `WITH_ONNX` build flag, producing five
additive rescoring features — `ms2_cosine`, `ms2_spectral_angle`, `ms2_pearson`,
`ms2_frac_pred_found`, `rt_abs_error` — that feed Percolator. Collision energy and
the RT transform are calibrated per run from confident PSMs. Reported at +3.7% and
+15.0% PSMs at 1% FDR. Disabled by default, and their CI does not exercise it yet.
The shape fits andes well: additive PIN columns are the change type this codebase
has repeatedly measured as safe, and andes already hands the PIN to Percolator.
This issue is about whether it pays here, and what it would cost.
## We are not starting from zero, which is the whole question
andes already writes seven intensity-derived features — `IntensitySignal`,
`FragPredExplained`, `FragPredChanceLLR`, `FragTopKObserved`, `RichIonLLR`,
`MeanMatchedIntensityRank`, `ModSiteIntensFrac` — and three RT features —
`DeltaRT`, `AbsDeltaRT`, `DeltaRTNorm`. Two GBDT ensembles already supply
fragment-intensity information from our own trained models.
The five OpenMS features land on ground we partly cover. Their uplift was measured
against their baseline, not ours. So the number that matters is the **marginal**
gain over andes's existing feature set, and nobody has measured that.
## Step 1 — measure before building anything (no Rust)
This can be settled without touching the engine:
1. Take an existing andes PIN from a benchmark set (Astral and UPS1 are the obvious
two, and both have entrapment databases).
2. Run the AlphaPeptDeep ONNX models offline, in Python, over exactly those PSMs.
3. Append the five columns to the PIN and re-run Percolator, five seeds, one
variable.
4. Report PSMs at 1% **and** true entrapment FDP, plus the learned Percolator
weights on the new columns and their correlation with our existing intensity and
RT features.
Outcomes: if the gain sits inside seed noise, or the new columns are collinear with
what we already emit, we stop and the issue closes with a measurement. If it clears
noise, we know the size of the prize before spending engineering on it.
**Contamination check is part of step 1, not an afterthought.** The bundled models
must not have been trained on the datasets we benchmark with. This project has been
bitten by exactly that (PXD014525 is in our own training ledger). Confirm the
AlphaPeptDeep training corpus against every benchmark accession before quoting any
number.
## Step 2 — if it pays, the engineering questions
- **Runtime.** `tract` is pure Rust and Apache-2.0, no C++ dependency, but narrower
operator coverage. `ort` binds ONNX Runtime (MIT), faster and broader, but it is a
native dependency. The packaging split we already decided — Apache core versus a
build carrying vendor dependencies — is the natural home for whichever we pick,
feature-gated and off by default.
- **Where inference happens is the whole speed risk.** It must run on the retained
top-N PSMs after the queue, never per candidate. andes scores on the order of
1e5–1e6 candidates per run; a transformer in that loop would erase the Comet
parity we just reached. This is the same lesson as the GBDT truncation work, where
the fix was to stop computing a model twice per candidate. Inference should also
be batched across spectra, because throughput not latency is the constraint.
- **Collision energy.** The models are NCE-conditioned. OpenMS fits it from
confident PSMs; we would need the same, and we already have a precursor
calibration pre-pass that establishes the pattern of learning a run-level
parameter from a confident subset.
- **Retention time.** Needs a run-level RT transform fitted from confident PSMs.
There is existing machinery to build on in the glyco RT offset code and the
DeltaRT features.
- **Licensing and provenance.** AlphaPeptDeep is Apache-2.0, but the *weights* need
their own confirmation, and shipping them means a NOTICE file, a per-file
statement of origin, and a note on whether the files are verbatim or derived. We
just went through this for the bundled pGlyco databases; the same paperwork
applies, and it matters more here given the model-independence goal.
- **Merge gate.** A new scoring input merges only if it beats current andes on
identifications or is faster. Same as everything else.
## What would kill it
A speed regression on the standard sets. Training-data overlap with our benchmarks.
A gain inside seed noise once our existing features are present. Or a dependency
that pushes the Apache-clean core further out of reach for a marginal return.
## Not proposed
Replacing our own trained models with these. The independence campaign exists for a
reason; this is about an additional, optional feature source, gated off by default,
exactly as OpenMS shipped it.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by taking an existing andes PIN from the Astral or UPS1 benchmark and run the AlphaPeptDeep ONNX models offline in Python. Append the five proposed columns, rerun Percolator with five seeds and one variable, and check 1% FDR PSMs, true entrapment FDP, weights, correlations, and training-data overlap. Done means reporting whether the marginal gain exceeds seed noise without contamination.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- backend, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100