snakemake / snakemake/snakemake
Idea: enable additional checksum types (md5) for `ensure`.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.9k
- Forks
- 656
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 9
Description
Is your feature request related to a problem? Please describe.
I just had a problem where my workflow crashed in a somewhat obscure way, and after some digging it turned out that one file was damaged. This file had been downloaded via the HTTP remote interface but somehow the file was corrupted (while the corresponding snakemake rule still succeeded).
Describe the solution you'd like
Luckily snakemake already has functionality to verify the checksum of output files with ensure! And Zenodo helpfully provides md5 checksums! But ensure only accepts a sha256 checksum...
Describe alternatives you've considered
Of course the alternative is to compute the sha256 checksum myself. But since a md5 checksum is often provided already, it would be nice to use that instead when available.
Additional context
It could also be worth mentioning the ensure functionality in "Remote files" section of the documentation! (Bonus points for also mentioning the retries keyword in the same breath.) I think verifying the checksum of a remote file should be a quite common need, but it actually took me some time to figure out that ensure is the right thing to use here. Of course, that's only the case if you write a rule just to download one or several files, and can use ensure on the output file(s) of this rule. Originally I was also thinking about whether it makes sense to introduction checksum verification functionality directly to e.g. remote.HTTP.
I'd be happy to have a go at implementing md5 checksums for ensure and open a pull request if this is desirable.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the existing ensure checksum handling and the documentation's "Remote files" section, both named in the issue. Check how checksum types are supported, then assess md5 verification for remote-file outputs and the requested documentation mention of ensure and retries. Done means md5 checksums work for ensure and the relevant documentation is updated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- devtools
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100