galaxyproject / galaxyproject/galaxy

Storing and Retrieving test data

Open
#7,900 6 comments 1 reaction 0 assignees View on GitHub
area/testing status/planning
Dominant language
Python
Stars
1.9k
Forks
1.2k
Avg merge
2d 16h
Merged PRs (30d)
169

Description

TL;DR: The way we store and retrieve tool (and maybe workflow) test data is not optimal.

The most prevalent approach today is to store the test data next to the tool wrappers in git, and then to upload the tool together with the test data to the tool shed. When installing a repo from the tool shed we also clone the test data.

This is problematic for a couple of reasons:
- The git and tool shed repository size can become unpractically large
- We need significant amount of storage for installed tools
- We're limited by the maximum test data size

There were some ideas and propositions in the past for storing test data:
- Allow referencing arbitrary URIs for fetching test data
- Provide test data on CVMFS, downloadable via HTTP at test time
- Have dedicated test data git repository

There are some considerations to each of them:
- Arbitrary URI:
- How do we ensure the test data is/stays available ?
- How do we know test data is immutable ?

- CVMFS: How do we get the data into CVMFS?
- Test data deposition via separate PR ?
- Deposit test data automatically via scanning existing tool shed repos ?
- Deposit test data with a bot that scans known galaxy tool github repos for test data ?

- Git test data repo
- Would have the same issues as storing test data next to tool IMO

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by tracing how tool wrappers store test data, how tools and data are uploaded to the Tool Shed, and how installed repositories clone it; compare the proposed URI, CVMFS, and dedicated-repository approaches. Done would require a decided and implemented storage strategy that addresses repository size, installation storage, availability, immutability, and maximum data-size limits.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.