cancervariants / cancervariants/gene-normalization

Investigate alternatives to gffutils in-memory data

Open
#264 3 comments 0 reactions 0 assignees View on GitHub
performance priority:medium technical debt
Dominant language
Python
Stars
3
Forks
4
PR merge metrics
No merged PRs in 30d

Description

I've been having some pretty crazy slowdown while trying to read the NCBI annotations into memory with `gffutils`. If this persists, there are at least two possible alternatives:

* Using `gffutils`, store data in a sqlite db (see below). We could also save this to a DB in the data folder and then check to see if it already exists so that we don't have to repeatedly create it. I think this will be pretty fast once the DB is created.

```
db = gffutils.create_db("gene/data/ncbi/ncbi_GRCh38.p14.gff", "tmp.db", force=True, merge_strategy="create_unique", keep_order=True)
```

* Investigate alternatives. `gffpandas` came up on a quick google, and it seems like a good fit. I bet we could spin our own up as well (gffpandas is, like, < 100 lines of actual code).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the code that reads gene/data/ncbi/ncbi_GRCh38.p14.gff and measure the current in-memory slowdown. Compare the proposed gffutils SQLite database in tmp.db with the gffpandas alternative, including whether a database saved in the data folder can be reused. Done means selecting and documenting a viable approach with evidence that repeated annotation loading is improved.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sqlite
Domain
bioinformatics, databases
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.