cancervariants / cancervariants/gene-normalization
Investigate alternatives to gffutils in-memory data
- Dominant language
- Python
- Stars
- 3
- Forks
- 4
- PR merge metrics
- No merged PRs in 30d
Description
I've been having some pretty crazy slowdown while trying to read the NCBI annotations into memory with `gffutils`. If this persists, there are at least two possible alternatives:
* Using `gffutils`, store data in a sqlite db (see below). We could also save this to a DB in the data folder and then check to see if it already exists so that we don't have to repeatedly create it. I think this will be pretty fast once the DB is created.
```
db = gffutils.create_db("gene/data/ncbi/ncbi_GRCh38.p14.gff", "tmp.db", force=True, merge_strategy="create_unique", keep_order=True)
```
* Investigate alternatives. `gffpandas` came up on a quick google, and it seems like a good fit. I bet we could spin our own up as well (gffpandas is, like, < 100 lines of actual code).
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the code that reads gene/data/ncbi/ncbi_GRCh38.p14.gff and measure the current in-memory slowdown. Compare the proposed gffutils SQLite database in tmp.db with the gffpandas alternative, including whether a database saved in the data folder can be reused. Done means selecting and documenting a viable approach with evidence that repeated annotation loading is improved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, sqlite
- Domain
- bioinformatics, databases
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100