galaxyproject / galaxyproject/brc-analytics
make our data less fluid
- Dominant language
- TypeScript
- Stars
- 7
- Forks
- 11
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 16
Description
sorry for that issue title, and the description thats about to follow.. i dont quite have a specific problem or solution yet, just a sense of one looming and some actions we might take that could potentially help that id like to put in peoples minds. here goes:
it seems to me we are too tightly bound to ncbi right now, to the point we effectively have no control of whats going to show on the site. it occasionally gets in the way when im working on something in the data catalog, but so far mostly in (a growing number of) small ways only. ill give an example of a few of these little issues though, and hopefully itll be clear how these can collectively add up to a real problem, particularly as we add more features and organisms.
1. they recently changed the ncbi taxon id that the covid assemblies are associated w. this means for us that the names of organisms will change, and their aliases, links to outbreaks weve defined will break, and the organism might even disappear from the site if i dont add the new tax id to our organisms yaml, etc unless i update all of that. its not a lot of work but i have no choice but to do that *right* now. i cant put it off until later, bc the second i run the data catalog for any even unrelated reason itll automatically update this.
- prev: https://api.ncbi.nlm.nih.gov/datasets/v2alpha/taxonomy/taxon/2697049/name_report
- new: https://api.ncbi.nlm.nih.gov/datasets/v2alpha/taxonomy/taxon/3418604/name_report
2. the name of c. auris changed and while everything still works and is even technically correct it happened silently without us even noticing at first and made an important organism difficult to find. (tangentially related, i think we need to stop trying to use the youth of the project as an excuse for things and instead just own it and go 'well look into it')
3. sometimes a new version of an assembly gets pushed out by ncbi, and then our qc report starts saying it cant find the old assembly on ncbi. but i dont think we want to be forced to update on their time tables. we should update, dont get me wrong, but we should have control over when we do that without losing organisms and assemblies from the site.
i think there are two things weve sort of already talked about that might help:
1. making the data catalog more modular, so we dont have to rerun everything constantly and can just rerun the bits we actually need. that will help somewhat sometimes, maybe push off the problem a bit. its not a real fix though, bc there can still be times when i need to rerun a piece that ends up updating things i didnt intend to focus on in that moment.
2. the db were talking about for the sra metadata could expand in scope to include everything. it probably should at some point, but i understand thats a big thing with a lot of implications and means a lot of work. at the very least, if we arent already i think we should plan the sra metadata stuffs w the understanding that itll likely eventually grow quite a lot beyond that use alone.
@nekrut @NoopDog i dont really think i can overstate how strongly i think this will eventually impact our ability to maintain the site long term. i just dont know what the words 'eventually' or 'long' actually mean yet in that sentence.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.