galaxyproject / galaxyproject/galaxy
Do not load datatypes registry during every metadata collection
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 1.2k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 169
Description
If we serialized just a mapping of extensions to metadata classes as JSON we wouldn't need to load that XML for every request and we wouldn't need to load the registry for datatypes which brings in every datatype class and every datatype Python dependency into memory. We could really optimize metadata collection - especially for smaller files, uploads with extensions set, etc.... This was a significant slow down for instance when running tool tests when I measured this years ago and there has been an explosion of datatypes and datatype dependencies since then.
This would also setup subsequent optimizations such as serializing just the mappings we need and further optimizing Galaxy's imports so that we don't import so much when running metadata collection. Also if we could just do this without loading models - maybe just writing out a JSON file that could be reloaded - that would be even better down the road. We shouldn't need sqlalchemy loaded to do metadata collection on the cluster.
Contributor guide
Research direction
Start by tracing metadata collection to identify where the XML and datatypes registry are loaded for each request. The issue describes a serialized JSON mapping of extensions to metadata classes as the intended direction; done would mean metadata collection avoids those repeated registry loads and preserves the stated optimization goal.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100