Optimise VCF import
- Dominant language
- Python
- Stars
- 30
- Forks
- 3
- Avg merge
- 9h 22m
- Merged PRs (30d)
- 40
Description
* At the moment, we write split VCFs then when complete, launch all of the jobs to process them. We should just launch them as soon as we write them.
* Upload step model should have an option to "delete input file when complete" eg if we know it's not used any more (to reduce temp disk usage during process)
* We only run annotation after the whole upload is complete. We should do this check after every insert of unknown variants, except make annotation range lock / pipeline have a minimum size, so we don't run inefficient small ones. Run the standard check with no minimum at the end of an import to catch any leftovers.
Contributor guide
No contributing guide indexed for this repository
Research direction
No files, tests, or entry points are named. Trace the VCF upload, split-file job launch, input-file cleanup, and unknown-variant annotation paths first; done means processing starts per written split, optional input deletion works, annotation runs incrementally with a minimum range and once at import completion for leftovers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- bioinformatics, data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100