snakemake / snakemake/snakemake

Idea: enable additional checksum types (md5) for `ensure`.

Open
#1,972 3 comments 6 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
2.9k
Forks
656
Avg merge
4d 12h
Merged PRs (30d)
9

Description

Is your feature request related to a problem? Please describe.
I just had a problem where my workflow crashed in a somewhat obscure way, and after some digging it turned out that one file was damaged. This file had been downloaded via the HTTP remote interface but somehow the file was corrupted (while the corresponding snakemake rule still succeeded).

Describe the solution you'd like
Luckily snakemake already has functionality to verify the checksum of output files with ensure! And Zenodo helpfully provides md5 checksums! But ensure only accepts a sha256 checksum...

Describe alternatives you've considered
Of course the alternative is to compute the sha256 checksum myself. But since a md5 checksum is often provided already, it would be nice to use that instead when available.

Additional context
It could also be worth mentioning the ensure functionality in "Remote files" section of the documentation! (Bonus points for also mentioning the retries keyword in the same breath.) I think verifying the checksum of a remote file should be a quite common need, but it actually took me some time to figure out that ensure is the right thing to use here. Of course, that's only the case if you write a rule just to download one or several files, and can use ensure on the output file(s) of this rule. Originally I was also thinking about whether it makes sense to introduction checksum verification functionality directly to e.g. remote.HTTP.

I'd be happy to have a go at implementing md5 checksums for ensure and open a pull request if this is desirable.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the existing ensure checksum handling and the documentation's "Remote files" section, both named in the issue. Check how checksum types are supported, then assess md5 verification for remote-file outputs and the requested documentation mention of ensure and retries. Done means md5 checksums work for ensure and the relevant documentation is updated.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devtools
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.