SACGF / SACGF/variantgrid

Analysis nodes: variant type, variant identifiers, classification/ClinVar filtering

Open
#1,789 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
30
Forks
3
Avg merge
9h 28m
Merged PRs (30d)
42

Description

🤖 Written by Claude

Plan: `claude/plans/analysis_filter_node_latent_demand_plan.md`

## Background

An audit of every `FilterNodeItem` on two production servers (raw output in `claude/prod_stats/`)
shows what users wanted an analysis node to do and hand-rolled a FilterNode for instead. SA Path
carries 46,686 filter nodes across 17,262 analyses; variantgrid.com carries 1,326 across 969.

Two distinct findings:

- **~95% of SA Path rules** are on fields no node exposes — `variant_class` (28,110 rules),
`dbsnp_rs_id` OR'd 4-7 deep (25,070), ClinVar `highest_pathogenicity != 1 AND != 2` (1,880).
- **~85% of variantgrid.com rules** re-implement options that already exist. 647 of 665
`cosmic_count` rules sit directly downstream of a BuiltInFilterNode that has `cosmic_count_min` on
its form; 329 of 358 `gnomad_hom_alt` rules sit directly downstream of a PopulationNode whose
`gnomad_hom_alt_max` already ORs in the null.

The work adds no new node classes — it extends three existing nodes and fixes stale data.

## Part A — Variant type on EffectNode

Variant class control on `DamageNode`, stored as an `ArrayField` of `VariantClass` values, appended
to `and_filters` in `_get_node_q()` with an include/exclude toggle. Editor groups the 39-member enum
(SNV / Indel / Copy number / Rearrangement / Mobile element / Repeat / Fusion / Other) with an
expander for individual classes.

**Manual testing:**
- [ ] SNV-only, indel-only, and exclude-SNV each return the expected variant counts
- [ ] Selecting nothing leaves the node's output unchanged
- [ ] The type restriction ANDs with damage scoring — a variant failing the damage criteria stays
filtered out regardless of its type
- [ ] Node chip shows the active type restriction on the canvas
- [ ] An EffectNode with only a type restriction set still reports `modifies_parents()`
- [ ] Analysis template export/import round-trips the selection
- [ ] Node clone (`save_clone`) carries the selection

## Part B — Free-text variant entry on IntersectionNode

Replaces the HGVS panel with a textarea, one entry per line, resolved at save time via the existing
`snpdb/search.py` receivers. Accepts dbSNP rsIDs, HGVS, variant coordinates (plain/VCF/gnomAD/
symbolic), loci, `chr:start-end` regions, ClinGen allele codes and COSMIC IDs. `CUSTOM_INTERVAL`
folds in; `SELECTED_INTERVALS`, `BACKEND_ENRICHMENT_KIT` and `CONTIG` stay as panels.

Resolution is a lookup — a line naming a variant absent from the database resolves to nothing and
says so, pointing at manual variant entry.

**Manual testing:**
- [ ] Each accepted entry format resolves to the right variant
- [ ] A multi-rsID `dbsnp_rs_id` row is found when searching for one of its IDs (the bug behind
19,174 prod rules using `equal` against a field VEP can pack with several rsIDs)
- [ ] Mixed formats in one paste all resolve
- [ ] An unresolvable line shows a per-line message and the node reports it via `get_warnings()`
- [ ] A `chr:start-end` region returns the same variants as the old CUSTOM_INTERVAL panel
- [ ] Existing analyses with HGVS and CUSTOM_INTERVAL nodes still work after the data migration
- [ ] Large paste crosses the VariantCollection threshold and returns the same result as a small one

## Part C — ClassificationsNode filter mode, with ClinVar

`node_input` field modelled on `TagNodeInput`: source (current behaviour, default for existing rows),
parent variants that ARE classified, parent variants that are NOT classified. `min_inputs` /
`max_inputs` become properties following the mode.

Adds ClinVar controls: clinical significance over the `highest_pathogenicity` scale with a separate
control for unscored records, review status / stars, conflicting interpretations, and variation ID.
`BuiltInFilterNode` keeps its ClinVar filter and node-count wiring.

**Manual testing:**
- [ ] Existing ClassificationsNodes still behave as sources with no parent
- [ ] Filter mode accepts a parent and narrows it; exclude mode returns the complement
- [ ] Connecting/disconnecting a parent respects the `min_inputs`/`max_inputs` for each mode
- [ ] ClinVar significance include/exclude gives the expected counts, and "exclude Benign + Likely
benign" keeps unscored variants (this is what 1,874 prod rules are asking for)
- [ ] Stars filter matches `BuiltInFilterNode` at the same threshold
- [ ] BuiltInFilterNode's ClinVar node counts are unaffected

## Part D — Data migration for stale column paths

Two `FilterNodeItem.field` values are orphaned copies of `VariantGridColumn.variant_column` paths
rewritten in `snpdb/migrations/0050_change_all_columns.py`. Both raise `FieldError` on load:

| Stored | Current | Prod rows |
|---|---|---|
| `clinvar__clinvar_review_status` | `clinvar__review_status` | 20 |
| `variantannotation__uniprot__tissue_specificity` | `variantannotation__transcript_version__gene_version__hgnc__uniprot__tissue_specificity` | 4 |

The migration also walks every remaining distinct `FilterNodeItem.field`, builds a queryset for each,
and logs any that raise — to tell whether this is a two-row cleanup or recurring drift.

Ships independently of A/B/C.

**Manual testing:**
- [ ] Analyses holding the two broken filters load instead of erroring
- [ ] Migration log lists any other unresolvable field paths
- [ ] `clinvar__review_status` rows whose `data` falls outside `ClinVarReviewStatus` are logged

## Follow-ups (discoverability, tracked separately)

- Surface `cosmic_count_min` in the BuiltInFilterNode editor when COSMIC is selected
- Give `PopulationNode.gnomad_hom_alt_max` more presence, and say it keeps rows with no gnomAD entry
- Name SpliceAI in EffectNode's splice control — 510 nodes rebuild
`impact = H/M OR spliceai_* >= 0.2`, which is `impact_min` + `splice_min`
- Single-gene shortcut on GeneListNode — 7,436 rules type a gene symbol into a filter node

Re-running `analysis_filter_audit.py` after release tells us whether shipped work absorbed its
demand. The IntersectionNode CONTIG panel (2026-06-11) is the natural first check — 19,115 rules
hand-rolled `chrom =` before it existed.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with claude/plans/analysis_filter_node_latent_demand_plan.md and inspect the existing DamageNode, EffectNode, IntersectionNode, ClassificationsNode, and BuiltInFilterNode implementations. Read snpdb/search.py for variant resolution and migration 0050_change_all_columns.py for stale paths. Done means the A–D manual checks pass, existing data remains usable, and migration logging identifies unresolved fields.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, databases, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.