dask / dask/cachey

Persistent cache with Zarr

Open
#7 24 comments 2 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
142
Forks
21
PR merge metrics
No merged PRs in 30d

Description

This is somewhat related to both #5 and #3 but slightly different. Basically I would like a persistent cache for use when working in a jupyter notebook. The motivation is very similar to https://github.com/rossant/ipycache i.e. if I restart a notebook I don't want to have to repeat any computations that previously finished. However I would like to use Zarr to store cache results not pickle because compression will save disk space. Also I would like to use a memoize function decorator rather than a cell magic, i.e., something more like the cachey memoize decorator and the [joblib Memory.cache decorator](https://pythonhosted.org/joblib/memory.html).

No problem if this is beyond scope for cachey but I thought I'd mention it in case there were any synergies with other requirements. On the technical side there are two main points to consider: one is how to generate a key from function arguments that is stable across python sessions (i.e., doesn't rely on Python's built-in hash function); the second is how to integrate with Zarr (or similar) for storage.

Contributor guide

Open the contributing guide

Research direction

Start with cachey's memoize decorator and the related requirements in issues #5 and #3, then consider how it would be used from a Jupyter notebook. Done means completed computations persist across Python sessions, keys remain stable for the same function arguments, and results are stored through Zarr rather than pickle.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook, python
Domain
tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.