somalier_existing_vcfs: batch ancestry across VCFs and fan extracts out to celery
- Dominant language
- Python
- Stars
- 30
- Forks
- 3
- Avg merge
- 9h 28m
- Merged PRs (30d)
- 42
Description
🤖 Written by Claude
`somalier_existing_vcfs` is very slow on tens of thousands of samples: it walks every VCF serially in-process and calls `somalier_vcf_id` for each, and ~95% of that per-VCF cost is `somalier ancestry` re-fitting on the 2,504 1kg background samples every time (the ~34s measured in #1147). At 10k VCFs that is ~100 hours, ~94 of them ancestry.
`somalier ancestry` takes any number of query files after `++`, and the fit cost is independent of how many there are, so the fix is:
1. **Batch ancestry** - one `somalier ancestry` call per batch of extracted samples, its TSV split back into per-VCF ancestry rows. 10k VCFs → a handful of calls → minutes.
2. **Fan extracts out to celery** - `somalier_vcf_extract` per VCF on `db_workers` instead of inline; the command waits, then batches ancestry, then runs the all-samples relate once.
The nightly all-samples relate is fine as is - somalier reports ~1 min for 4,500 samples and drops unrelated pairs by default.
Plan: [`claude/plans/1850_somalier_batch_ancestry_plan.md`](https://github.com/SACGF/variantgrid/blob/master/claude/plans/1850_somalier_batch_ancestry_plan.md)
Related: #1147 (why ancestry is the slow stage), #1842 (skip ancestry/relate below a minimum number of sites).
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with claude/plans/1850_somalier_batch_ancestry_plan.md and the somalier_existing_vcfs entry point; trace how somalier_vcf_id and somalier_vcf_extract are currently invoked. Confirm that extraction fans out to db_workers, ancestry runs once per batch, per-VCF ancestry rows are restored, and the all-samples relate runs once after completion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, bioinformatics, distributed-systems
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100