galaxyproject / galaxyproject/planemo
Uploading/mirroring of test data
- Dominant language
- Python
- Stars
- 110
- Forks
- 102
- Avg merge
- 4d 21h
- Merged PRs (30d)
- 13
Description
I would like to be able to have a command that is essentially
`planemo upload_test_data` that will take local files or URLs from a tool or workflow test, upload them to zenodo if they don’t exist yet and rewrite the test files to point at the newly created (or already existing) test data.
Complications are:
- there’s no /api/whoami, so we don’t know the user id (if we don’t ask for it …)
- there doesn’t seem to be a reliable way to list records of a given user, only a free-form search, so users would also have to provide the record (which is OK ?)
- there is no search for file hash sums which would have allowed us to easily check if the file already exists (but we can iterate over the files of a record and check if any file with the same checksum exists)
Maybe those are not dealbreakers for uploads to zenodo.
Then finally there is the question of whether or not it is something that would be frowned upon by zenodo
if every update that touches test data generates a new DOI (I’m a little unclear about the cost of updating a record, this might not count as minting a new DOI ?).
Bjoern proposed that we could instead use [depot.galaxyproject.org](http://depot.galaxyproject.org/), though that of course rules out users doing the upload themselves prior to merging and it’s questionable if we can and want to provide this service to all communities as well.
Another option is that we have different targets for `upload_test_data`, where one could just be any public galaxy instance (need to publish the history or the datasets in that case). Only when we merge the PR we actually upload data to zenodo, mint a DOI and replace the location in the test file.
Contributor guide
Research direction
The issue names the proposed Planemo CLI command `upload_test_data`, tool and workflow test files, Zenodo, and depot.galaxyproject.org, but no implementation files or tests. Start by reviewing existing Planemo CLI commands and test-data handling, then resolve the upload target, identity, duplicate detection, DOI, and rewrite behavior. Done means the chosen scope is implemented with coverage for the supported upload flow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100