Persistent cache with Zarr
- Dominant language
- Python
- Stars
- 142
- Forks
- 21
- PR merge metrics
- No merged PRs in 30d
Description
This is somewhat related to both #5 and #3 but slightly different. Basically I would like a persistent cache for use when working in a jupyter notebook. The motivation is very similar to https://github.com/rossant/ipycache i.e. if I restart a notebook I don't want to have to repeat any computations that previously finished. However I would like to use Zarr to store cache results not pickle because compression will save disk space. Also I would like to use a memoize function decorator rather than a cell magic, i.e., something more like the cachey memoize decorator and the [joblib Memory.cache decorator](https://pythonhosted.org/joblib/memory.html).
No problem if this is beyond scope for cachey but I thought I'd mention it in case there were any synergies with other requirements. On the technical side there are two main points to consider: one is how to generate a key from function arguments that is stable across python sessions (i.e., doesn't rely on Python's built-in hash function); the second is how to integrate with Zarr (or similar) for storage.
Contributor guide
Research direction
Start with cachey's memoize decorator and the related requirements in issues #5 and #3, then consider how it would be used from a Jupyter notebook. Done means completed computations persist across Python sessions, keys remain stable for the same function arguments, and results are stored through Zarr rather than pickle.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter-notebook, python
- Domain
- tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100