SACGF / SACGF/variantgrid

Somalier - skip ancestry/relate for VCFs below a minimum number of genotyped sites

Open
#1,842 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
30
Forks
3
Avg merge
9h 28m
Merged PRs (30d)
42

Description

🤖 Written by Claude

Split out of #1147, which was about somalier taking 43% of upload time. That headline problem is gone — the somalier reference data moved off EFS to local disk on vgaws on 2026-07-13 and the step went from a median of 202s to 33s (benchmarks in #1147). This issue covers the remaining ~30s.

### What's left

A full `somalier_vcf_id` run is ~33s in steady state. Almost all of that is `somalier ancestry`, which re-reads the 2,504 1kg `.somalier` files on every run regardless of how much data the VCF actually has. VCFs with no genotype already return early and take ~3s, so the ~30s gap is entirely ancestry + relate on VCFs that may not have enough sites to give a meaningful answer.

### Proposal

Add a minimum genotyped-sites threshold to `settings.SOMALIER`, and skip ancestry and relate when a VCF's extract falls below it. Somalier's own guidance is that relatedness and ancestry estimates need a reasonable number of the ~17.5k sites covered; a gene panel or a heavily filtered VCF can come in with very few and the result is noise we then store and display.

Points to settle:

- Where the count comes from — `somalier extract` writes the number of sites to the `.somalier` file, so the threshold can be applied after extract (cheap) and before ancestry (expensive). That ordering is what makes this worth doing.
- What the threshold should be. Worth measuring the site counts across existing `SomalierVCFExtract` records on vgaws before picking a number.
- How a skip is recorded — `ProcessingStatus.SKIPPED` on the extract mirrors the existing no-genotype path, so the UI can say why there's no ancestry rather than showing a gap.

### Related

An `ancestry_enabled` setting was also proposed in #1147 as a way to turn the expensive stage off wholesale. That's a blunter version of the same lever; a site-count threshold is probably the better default, so noting it here rather than opening a third issue.

Relevant code: `snpdb/tasks/somalier_tasks.py` (`somalier_vcf_id`, `_somalier_ancestry`, `_somalier_relate`), `snpdb/models/models_somalier.py` (`SomalierConfig`, `SomalierVCFExtract`).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in snpdb/tasks/somalier_tasks.py at somalier_vcf_id, _somalier_ancestry, and _somalier_relate, then inspect SomalierConfig and SomalierVCFExtract in snpdb/models/models_somalier.py. Review the existing no-genotype early-return path and measure site counts across existing records on vgaws before settling the threshold. Done means low-site extracts skip ancestry and relate with ProcessingStatus.SKIPPED, while sufficient-site VCFs retain the current flow.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.