galaxyproject / galaxyproject/pulsar
Caching - Rewrite
- Dominant language
- Python
- Stars
- 46
- Forks
- 62
- Avg merge
- 3d 15h
- Merged PRs (30d)
- 14
Description
Vaguely speaking all caching is currently path based - this is quite wrong. It makes too many assumptions on the Galaxy side and is inflexible given all the crazy stuff Galaxy or the object store may choose to do with paths. It would be better to change the assumptions to this
- Every file in Galaxy's object store should be considered immutable - while Galaxy doesn't enforce this directly - no well behaved tool should be modifying these files.
- Every dataset in Galaxy should have a UUID (I believe they do thanks to @kellrott).
Together these assumptions mean the UUID should uniquely describe a file.
- [ ] Include the UUID information for datasets in the job submission request. Pulsar job runner on Galaxy-side should have access to the HDAs and so `hda.dataset.uuid` should be populated. This information has to be passed down through the client layer and included the request object... somehow.
- [ ] Rework Pulsar's caching layer to allow caching by UUID instead of path (may not actually require changes).
- [ ] Use cache during preprocessing of the job. Likely this means reworking Pulsar's `FileActionMapper` or perhaps the layer above that uses it - to check the cache for files and stage them to the path if needed. There are different things that could be done here - I am unsure which is best - it could do it transparently and then return a `NoneAction`, it could return a `GetFromCache` action, maybe something else.
- [ ] Use cache during postprocessing of the job. Should again involve the `FileActionMapper` or something above it. In this case though - the original action needs to be performed and the file should be injected into the cache using the UUID.
Contributor guide
Assessment
This issue has not been assessed yet.