galaxyproject / galaxyproject/galaxy

Content hash for workflows

Open
#17,536 1 comment 2 reactions 0 assignees View on GitHub
area/workflows kind/enhancement
Dominant language
Python
Stars
1.9k
Forks
1.2k
Avg merge
2d 23h
Merged PRs (30d)
158

Description

It would be great if we could find remote workflows based on their content hash. Applications of this are to prevent repeated upload of workflows via `planemo run` or `planemo test`. Not having duplicates of workflows (or at least being able to say
"these are the same workflows") would make it simpler to query for invocations of a workflow that have already been executed.

To some degree the "it's the same workflow" case is covered by TRS ids, but these are not available for all workflows, and certainly not during development.
We also have UUIDs associated with Workflows, but these are not at all guaranteed to actually be unique on a given Galaxy instance, and anyone is free to edit the workflow without updating the UUID.

@jmchilton suggested that we could cover the planemo case with a dotfile next to the workflow, but that would be relatively complicated for workflow runs to be triggered by a CI.

My suggestion would be to take the original document as uploaded and compute the md5 for that document.
When modifying a workflow we could serialize the workflow once and calculate an md5. I would store that checksum on the Workflow model and not the StoredWorkflow model.

The downside to this approach is that any change in formatting of the document, or change in serialization would break matching the hash if the workflow is downloaded after we have calculated and stored the checksum, at a time when we did actually change something about the workflow serialization.
To address this, we could calculate the content-hash whenever a workflow is downloaded and store it as a valid (but not neccessarily) unique hash for a workflow. This should even cover the case of switching from a .ga representation to a gxformat2 version of a workflow

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.