galaxyproject / galaxyproject/galaxy
Storing and Retrieving test data
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 1.2k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 169
Description
TL;DR: The way we store and retrieve tool (and maybe workflow) test data is not optimal.
The most prevalent approach today is to store the test data next to the tool wrappers in git, and then to upload the tool together with the test data to the tool shed. When installing a repo from the tool shed we also clone the test data.
This is problematic for a couple of reasons:
- The git and tool shed repository size can become unpractically large
- We need significant amount of storage for installed tools
- We're limited by the maximum test data size
There were some ideas and propositions in the past for storing test data:
- Allow referencing arbitrary URIs for fetching test data
- Provide test data on CVMFS, downloadable via HTTP at test time
- Have dedicated test data git repository
There are some considerations to each of them:
- Arbitrary URI:
- How do we ensure the test data is/stays available ?
- How do we know test data is immutable ?
- CVMFS: How do we get the data into CVMFS?
- Test data deposition via separate PR ?
- Deposit test data automatically via scanning existing tool shed repos ?
- Deposit test data with a bot that scans known galaxy tool github repos for test data ?
- Git test data repo
- Would have the same issues as storing test data next to tool IMO
Contributor guide
Research direction
No files, tests, or entry points are named. Start by tracing how tool wrappers store test data, how tools and data are uploaded to the Tool Shed, and how installed repositories clone it; compare the proposed URI, CVMFS, and dedicated-repository approaches. Done would require a decided and implemented storage strategy that addresses repository size, installation storage, availability, immutability, and maximum data-size limits.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100