SACGF / SACGF/variantgrid

Optimise VCF import

Closed
#67 3 comments 0 reactions 0 assignees View on GitHub
VCF
Dominant language
Python
Stars
30
Forks
3
Avg merge
9h 22m
Merged PRs (30d)
40

Description

* At the moment, we write split VCFs then when complete, launch all of the jobs to process them. We should just launch them as soon as we write them.
* Upload step model should have an option to "delete input file when complete" eg if we know it's not used any more (to reduce temp disk usage during process)
* We only run annotation after the whole upload is complete. We should do this check after every insert of unknown variants, except make annotation range lock / pipeline have a minimum size, so we don't run inefficient small ones. Run the standard check with no minimum at the end of an import to catch any leftovers.

Contributor guide

No contributing guide indexed for this repository

Research direction

No files, tests, or entry points are named. Trace the VCF upload, split-file job launch, input-file cleanup, and unknown-variant annotation paths first; done means processing starts per written split, optional input deletion works, annotation runs incrementally with a minimum range and once at import completion for leftovers.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
bioinformatics, data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.