SACGF / SACGF/variantgrid

somalier_existing_vcfs: batch ancestry across VCFs and fan extracts out to celery

Open
#1,850 0 comments 0 reactions 0 assignees View on GitHub
Has Plan
Dominant language
Python
Stars
30
Forks
3
Avg merge
9h 28m
Merged PRs (30d)
42

Description

🤖 Written by Claude

`somalier_existing_vcfs` is very slow on tens of thousands of samples: it walks every VCF serially in-process and calls `somalier_vcf_id` for each, and ~95% of that per-VCF cost is `somalier ancestry` re-fitting on the 2,504 1kg background samples every time (the ~34s measured in #1147). At 10k VCFs that is ~100 hours, ~94 of them ancestry.

`somalier ancestry` takes any number of query files after `++`, and the fit cost is independent of how many there are, so the fix is:

1. **Batch ancestry** - one `somalier ancestry` call per batch of extracted samples, its TSV split back into per-VCF ancestry rows. 10k VCFs → a handful of calls → minutes.
2. **Fan extracts out to celery** - `somalier_vcf_extract` per VCF on `db_workers` instead of inline; the command waits, then batches ancestry, then runs the all-samples relate once.

The nightly all-samples relate is fine as is - somalier reports ~1 min for 4,500 samples and drops unrelated pairs by default.

Plan: [`claude/plans/1850_somalier_batch_ancestry_plan.md`](https://github.com/SACGF/variantgrid/blob/master/claude/plans/1850_somalier_batch_ancestry_plan.md)

Related: #1147 (why ancestry is the slow stage), #1842 (skip ancestry/relate below a minimum number of sites).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with claude/plans/1850_somalier_batch_ancestry_plan.md and the somalier_existing_vcfs entry point; trace how somalier_vcf_id and somalier_vcf_extract are currently invoked. Confirm that extraction fans out to db_workers, ancestry runs once per batch, per-VCF ancestry rows are restored, and the all-samples relate runs once after completion.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, bioinformatics, distributed-systems
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.