Somalier - skip ancestry/relate for VCFs below a minimum number of genotyped sites
- Dominant language
- Python
- Stars
- 30
- Forks
- 3
- Avg merge
- 9h 28m
- Merged PRs (30d)
- 42
Description
🤖 Written by Claude
Split out of #1147, which was about somalier taking 43% of upload time. That headline problem is gone — the somalier reference data moved off EFS to local disk on vgaws on 2026-07-13 and the step went from a median of 202s to 33s (benchmarks in #1147). This issue covers the remaining ~30s.
### What's left
A full `somalier_vcf_id` run is ~33s in steady state. Almost all of that is `somalier ancestry`, which re-reads the 2,504 1kg `.somalier` files on every run regardless of how much data the VCF actually has. VCFs with no genotype already return early and take ~3s, so the ~30s gap is entirely ancestry + relate on VCFs that may not have enough sites to give a meaningful answer.
### Proposal
Add a minimum genotyped-sites threshold to `settings.SOMALIER`, and skip ancestry and relate when a VCF's extract falls below it. Somalier's own guidance is that relatedness and ancestry estimates need a reasonable number of the ~17.5k sites covered; a gene panel or a heavily filtered VCF can come in with very few and the result is noise we then store and display.
Points to settle:
- Where the count comes from — `somalier extract` writes the number of sites to the `.somalier` file, so the threshold can be applied after extract (cheap) and before ancestry (expensive). That ordering is what makes this worth doing.
- What the threshold should be. Worth measuring the site counts across existing `SomalierVCFExtract` records on vgaws before picking a number.
- How a skip is recorded — `ProcessingStatus.SKIPPED` on the extract mirrors the existing no-genotype path, so the UI can say why there's no ancestry rather than showing a gap.
### Related
An `ancestry_enabled` setting was also proposed in #1147 as a way to turn the expensive stage off wholesale. That's a blunter version of the same lever; a site-count threshold is probably the better default, so noting it here rather than opening a third issue.
Relevant code: `snpdb/tasks/somalier_tasks.py` (`somalier_vcf_id`, `_somalier_ancestry`, `_somalier_relate`), `snpdb/models/models_somalier.py` (`SomalierConfig`, `SomalierVCFExtract`).
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in snpdb/tasks/somalier_tasks.py at somalier_vcf_id, _somalier_ancestry, and _somalier_relate, then inspect SomalierConfig and SomalierVCFExtract in snpdb/models/models_somalier.py. Review the existing no-genotype early-return path and measure site counts across existing records on vgaws before settling the threshold. Done means low-site extracts skip ancestry and relate with ProcessingStatus.SKIPPED, while sufficient-site VCFs retain the current flow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100