galaxyproject / galaxyproject/galaxy

Do not load datatypes registry during every metadata collection

Open
#4,609 2 comments 2 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.9k
Forks
1.2k
Avg merge
2d 16h
Merged PRs (30d)
169

Description

If we serialized just a mapping of extensions to metadata classes as JSON we wouldn't need to load that XML for every request and we wouldn't need to load the registry for datatypes which brings in every datatype class and every datatype Python dependency into memory. We could really optimize metadata collection - especially for smaller files, uploads with extensions set, etc.... This was a significant slow down for instance when running tool tests when I measured this years ago and there has been an explosion of datatypes and datatype dependencies since then.

This would also setup subsequent optimizations such as serializing just the mappings we need and further optimizing Galaxy's imports so that we don't import so much when running metadata collection. Also if we could just do this without loading models - maybe just writing out a JSON file that could be reloaded - that would be even better down the road. We shouldn't need sqlalchemy loaded to do metadata collection on the cluster.

Contributor guide

Open the contributing guide

Research direction

Start by tracing metadata collection to identify where the XML and datatypes registry are loaded for each request. The issue describes a serialized JSON mapping of extensions to metadata classes as the intended direction; done would mean metadata collection avoids those repeated registry loads and preserves the stated optimization goal.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.