Streamline VCF annotator ABC definition
- Dominant language
- Python
- Stars
- 62
- Forks
- 42
- Avg merge
- 1h 2m
- Merged PRs (30d)
- 1
Description
In #502 I defined an abstract VCF annotation interface where implementations could produce side-effects on a per-variant basis as well as upon the entire collection of variants. Of course, to do the latter, you need to maintain a collection of all the variants for the duration of annotation. Since no side effects are defined on the base class, I defined a class variable that needs to be switched on to enable collection-based operations, in addition to defining the hook function itself.
This of course becomes a problem for large VCFs, because Pydantic (especially v2) objects occupy a lot of memory. And it's a problem that this value is defined at a class level even though inputs are given on a per-annotation basis, so you might not even want to be doing the collection. In #545 I added an additional, optionally-implementable method for implementations to conditionally suppress that collection. I think this interface/related operations could be improved in a future major release (+ some smaller possible improvements in the meantime)
* Check for collection suppression on a per-annotation basis. Ditch the class variable, just redefine the `should_collect_alleles` method.
* The allele collection should contain serialized objects, rather than the full Pydantic object.
* NDJSON should be written on a per-variation basis rather than being added at the end.
* Maybe don't support dumping as a PKL (it's limited to just re-loading the Python object, it theoretically raises security issues). Maybe we shouldn't be supporting this whole "perform on a collection" hook at all, insofar as non-small VCFs are going to be expensive to hold variants in memory for. Maybe it's worth reorienting our output framework on things that can be streamed to.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.