SACGF / SACGF/variantgrid

Query optimisation: planner-hostile joins and unmaterialised subqueries

Open
#1,722 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
30
Forks
3
Avg merge
9h 28m
Merged PRs (30d)
42

Description

🤖 Written by Claude.

Follow-up sweep after #1720. That issue found that joining the 17M-row `Variant` table through the small fixed `GenomeBuildContig` table collapsed the planner's row estimate to 1, giving a nested loop that re-scanned the variant side once per contig (83s -> 6s once rewritten as an IN list).

The same shape — a huge table filtered through a small lookup table, or through an unmaterialised subquery — appears in several other places. In each case the small side can be resolved to a list of ids in Python first.

### Planner-hostile joins

**`Variant.qs_from_variant_coordinate`** (`snpdb/models/models_variant.py:728`) still filters via `locus__contig__genomebuildcontig__genome_build` plus a `locus__contig__name` string comparison. The contig is fully determined in Python by `GenomeBuild.chrom_contig_mappings` (a cached dict that already handles name/UCSC/accession aliases), so this can filter `locus__contig=contig` directly and land on the leading edge of the `snpdb_locus(contig_id, position, ref_id)` unique index with no lookup joins at all.

This is the single-variant-by-coordinate lookup behind `Variant.get_from_variant_coordinate`, so it runs per variant during HGVS resolution (`genes/hgvs/hgvs_matcher.py:721`), ClinGen allele registration (`snpdb/clingen_allele.py:312,418`) and wiki import.

**Variant search chrom matching** (`snpdb/signals/variant_search.py:277`) uses `Variant.get_chrom_q`, an OR of two `__iexact` comparisons against the joined `Contig` table, on every search-box coordinate lookup. The genome build is in hand, so the same contig resolution applies.

**OMIM built-in filter** (`analysis/models/nodes/node_counts.py:17`) is `Q(variantannotation__gene__geneannotation__omim_terms__isnull=False)` — a three-hop join off a 17M-row table into a small lookup, with no `version=` filter. `GeneAnnotation` is `unique_together ("version", "gene")`, so this also matches rows from every `GeneAnnotationVersion` rather than the current one. Runs for node counts on every analysis node.

### Unmaterialised subqueries

The codebase already establishes the fix for these twice — `analysis/models/nodes/filters/tag_node.py:52` ("Tagging is done manually so this will only ever be small - much faster to convert to list") and `classification/models/classification.py:2112` ("List is much faster than inner query"). These sites were missed:

- `VariantTag.variants_for_build_q` (`analysis/models/models_variant_tag.py:111`) — hot via TagNode on every analysis re-run and via the TaggedVariantGrid page.
- `get_has_variant_tags` (`snpdb/variant_queries.py:42`) and the gene symbol grid tag filter (`genes/grids.py:169`) — same shape.
- `get_has_classifications_q` (`snpdb/variant_queries.py:34`) round-trips `VariantAllele -> Allele -> VariantAllele -> Variant -> Classification`. `Classification` has an `allele` FK, so the allele ids can be resolved directly.

### Repeated and oversized queries

- `snpdb/views/datatable_view.py:573,579` runs the same permission-scoped `COUNT(*)` twice on every DataTable draw whenever no filter parameters are supplied (`filter_queryset` returns the queryset unchanged).
- `Variant.get_best_variant_transcript_annotation` (`snpdb/models/models_variant.py:904`) issues two queries against `VariantTranscriptAnnotation` that differ only by `canonical`.
- `analysis/models/nodes/filters/filter_node.py:22,37` uses `.count()` purely as a boolean.
- `classification/views/views.py:981` fetches whole `Classification` rows (including the wide `evidence` JSONB) to read `lab` and `created`.
- `classification/views/clinvar_export_view.py:606` counts the same queryset twice; `beacon/datasets.py:167` counts then iterates the same queryset.

### Checked and cleared

`snpdb/variants_to_vcf.py` already matches the fixed `write_qs_to_vcf` shape from #1720. The analysis node `contig_ids` joins drive tiny per-node tables. `.order_by(...).count()` sites are fine — Django clears ordering for un-sliced counts.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the named entry points in snpdb/models/models_variant.py, snpdb/variant_queries.py, analysis/models/models_variant_tag.py, and the listed view and filter files; compare them with the established list-conversion examples in tag_node.py and classification.py. Trace the affected query paths and use query plans or existing performance tests to verify that redundant counts, wide fetches, joins, and subqueries are reduced without changing results.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, databases, performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.