Analysis nodes: variant type, variant identifiers, classification/ClinVar filtering
- Dominant language
- Python
- Stars
- 30
- Forks
- 3
- Avg merge
- 9h 28m
- Merged PRs (30d)
- 42
Description
🤖 Written by Claude
Plan: `claude/plans/analysis_filter_node_latent_demand_plan.md`
## Background
An audit of every `FilterNodeItem` on two production servers (raw output in `claude/prod_stats/`)
shows what users wanted an analysis node to do and hand-rolled a FilterNode for instead. SA Path
carries 46,686 filter nodes across 17,262 analyses; variantgrid.com carries 1,326 across 969.
Two distinct findings:
- **~95% of SA Path rules** are on fields no node exposes — `variant_class` (28,110 rules),
`dbsnp_rs_id` OR'd 4-7 deep (25,070), ClinVar `highest_pathogenicity != 1 AND != 2` (1,880).
- **~85% of variantgrid.com rules** re-implement options that already exist. 647 of 665
`cosmic_count` rules sit directly downstream of a BuiltInFilterNode that has `cosmic_count_min` on
its form; 329 of 358 `gnomad_hom_alt` rules sit directly downstream of a PopulationNode whose
`gnomad_hom_alt_max` already ORs in the null.
The work adds no new node classes — it extends three existing nodes and fixes stale data.
## Part A — Variant type on EffectNode
Variant class control on `DamageNode`, stored as an `ArrayField` of `VariantClass` values, appended
to `and_filters` in `_get_node_q()` with an include/exclude toggle. Editor groups the 39-member enum
(SNV / Indel / Copy number / Rearrangement / Mobile element / Repeat / Fusion / Other) with an
expander for individual classes.
**Manual testing:**
- [ ] SNV-only, indel-only, and exclude-SNV each return the expected variant counts
- [ ] Selecting nothing leaves the node's output unchanged
- [ ] The type restriction ANDs with damage scoring — a variant failing the damage criteria stays
filtered out regardless of its type
- [ ] Node chip shows the active type restriction on the canvas
- [ ] An EffectNode with only a type restriction set still reports `modifies_parents()`
- [ ] Analysis template export/import round-trips the selection
- [ ] Node clone (`save_clone`) carries the selection
## Part B — Free-text variant entry on IntersectionNode
Replaces the HGVS panel with a textarea, one entry per line, resolved at save time via the existing
`snpdb/search.py` receivers. Accepts dbSNP rsIDs, HGVS, variant coordinates (plain/VCF/gnomAD/
symbolic), loci, `chr:start-end` regions, ClinGen allele codes and COSMIC IDs. `CUSTOM_INTERVAL`
folds in; `SELECTED_INTERVALS`, `BACKEND_ENRICHMENT_KIT` and `CONTIG` stay as panels.
Resolution is a lookup — a line naming a variant absent from the database resolves to nothing and
says so, pointing at manual variant entry.
**Manual testing:**
- [ ] Each accepted entry format resolves to the right variant
- [ ] A multi-rsID `dbsnp_rs_id` row is found when searching for one of its IDs (the bug behind
19,174 prod rules using `equal` against a field VEP can pack with several rsIDs)
- [ ] Mixed formats in one paste all resolve
- [ ] An unresolvable line shows a per-line message and the node reports it via `get_warnings()`
- [ ] A `chr:start-end` region returns the same variants as the old CUSTOM_INTERVAL panel
- [ ] Existing analyses with HGVS and CUSTOM_INTERVAL nodes still work after the data migration
- [ ] Large paste crosses the VariantCollection threshold and returns the same result as a small one
## Part C — ClassificationsNode filter mode, with ClinVar
`node_input` field modelled on `TagNodeInput`: source (current behaviour, default for existing rows),
parent variants that ARE classified, parent variants that are NOT classified. `min_inputs` /
`max_inputs` become properties following the mode.
Adds ClinVar controls: clinical significance over the `highest_pathogenicity` scale with a separate
control for unscored records, review status / stars, conflicting interpretations, and variation ID.
`BuiltInFilterNode` keeps its ClinVar filter and node-count wiring.
**Manual testing:**
- [ ] Existing ClassificationsNodes still behave as sources with no parent
- [ ] Filter mode accepts a parent and narrows it; exclude mode returns the complement
- [ ] Connecting/disconnecting a parent respects the `min_inputs`/`max_inputs` for each mode
- [ ] ClinVar significance include/exclude gives the expected counts, and "exclude Benign + Likely
benign" keeps unscored variants (this is what 1,874 prod rules are asking for)
- [ ] Stars filter matches `BuiltInFilterNode` at the same threshold
- [ ] BuiltInFilterNode's ClinVar node counts are unaffected
## Part D — Data migration for stale column paths
Two `FilterNodeItem.field` values are orphaned copies of `VariantGridColumn.variant_column` paths
rewritten in `snpdb/migrations/0050_change_all_columns.py`. Both raise `FieldError` on load:
| Stored | Current | Prod rows |
|---|---|---|
| `clinvar__clinvar_review_status` | `clinvar__review_status` | 20 |
| `variantannotation__uniprot__tissue_specificity` | `variantannotation__transcript_version__gene_version__hgnc__uniprot__tissue_specificity` | 4 |
The migration also walks every remaining distinct `FilterNodeItem.field`, builds a queryset for each,
and logs any that raise — to tell whether this is a two-row cleanup or recurring drift.
Ships independently of A/B/C.
**Manual testing:**
- [ ] Analyses holding the two broken filters load instead of erroring
- [ ] Migration log lists any other unresolvable field paths
- [ ] `clinvar__review_status` rows whose `data` falls outside `ClinVarReviewStatus` are logged
## Follow-ups (discoverability, tracked separately)
- Surface `cosmic_count_min` in the BuiltInFilterNode editor when COSMIC is selected
- Give `PopulationNode.gnomad_hom_alt_max` more presence, and say it keeps rows with no gnomAD entry
- Name SpliceAI in EffectNode's splice control — 510 nodes rebuild
`impact = H/M OR spliceai_* >= 0.2`, which is `impact_min` + `splice_min`
- Single-gene shortcut on GeneListNode — 7,436 rules type a gene symbol into a filter node
Re-running `analysis_filter_audit.py` after release tells us whether shipped work absorbed its
demand. The IntersectionNode CONTIG panel (2026-06-11) is the natural first check — 19,115 rules
hand-rolled `chrom =` before it existed.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with claude/plans/analysis_filter_node_latent_demand_plan.md and inspect the existing DamageNode, EffectNode, IntersectionNode, ClassificationsNode, and BuiltInFilterNode implementations. Read snpdb/search.py for variant resolution and migration 0050_change_all_columns.py for stale paths. Done means the A–D manual checks pass, existing data remains usable, and migration logging identifies unresolved fields.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, databases, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100