galaxyproject / galaxyproject/pulsar

Caching - Rewrite

Open
#48 0 comments 0 reactions 1 assignee Claimed by @hexylena View on GitHub
enhancement
Dominant language
Python
Stars
46
Forks
62
Avg merge
3d 15h
Merged PRs (30d)
14

Description

Vaguely speaking all caching is currently path based - this is quite wrong. It makes too many assumptions on the Galaxy side and is inflexible given all the crazy stuff Galaxy or the object store may choose to do with paths. It would be better to change the assumptions to this
- Every file in Galaxy's object store should be considered immutable - while Galaxy doesn't enforce this directly - no well behaved tool should be modifying these files.
- Every dataset in Galaxy should have a UUID (I believe they do thanks to @kellrott).

Together these assumptions mean the UUID should uniquely describe a file.
- [ ] Include the UUID information for datasets in the job submission request. Pulsar job runner on Galaxy-side should have access to the HDAs and so `hda.dataset.uuid` should be populated. This information has to be passed down through the client layer and included the request object... somehow.
- [ ] Rework Pulsar's caching layer to allow caching by UUID instead of path (may not actually require changes).
- [ ] Use cache during preprocessing of the job. Likely this means reworking Pulsar's `FileActionMapper` or perhaps the layer above that uses it - to check the cache for files and stage them to the path if needed. There are different things that could be done here - I am unsure which is best - it could do it transparently and then return a `NoneAction`, it could return a `GetFromCache` action, maybe something else.
- [ ] Use cache during postprocessing of the job. Should again involve the `FileActionMapper` or something above it. In this case though - the original action needs to be performed and the file should be injected into the cache using the UUID.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.