con / con/fscacher

Next design, directories oriented?/localized, is needed

Open
#69 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2
Forks
2
PR merge metrics
No merged PRs in 30d

Description

This issue could be considered a duplicate of earlier https://github.com/dandi/dandi-cli/issues/848#issuecomment-988055068 but as that one just started with suggestion of an alternative implementation within dandi-cli, I decided to file a separate one within fscacher which would provide more coverage over the situation and since I still hope that fscacher could be a reusable package to provide needed solution.

Outstanding issues we have which are all bottlenecking IMHO in our initial simplistic fscacher implementation, which

- `I1`: uses the same cache for a function (or collection of functions) regardless of their parametrization etc
- `I2`: places all caches into centralized user-wide cache directory (e.g., `~/.cache/fscacher/{cachename}`): caches related to a dataset which might get removed,
- `I3`: uses joblib's memoize so that each invocation gets its own directory + 2 files on disk (thus 3 inodes):

```shell
(dandi-devel) jovyan@jupyter-yarikoptic:~/.cache/fscacher/dandi-checksums/joblib/dandi/support/digests$
$ ls get_zarr_checksum/b74a957ae8f7e2e4fc2b07d1aaa73775/
metadata.json output.pkl
```
- in effect allows to avoid need for any locking since fingerprint defines the directory name to use

Such simplistic initial design showed its limitations by
- making difficult to prune old cache selectively (helping interfacing/functionality is proposed in https://github.com/joblib/joblib/pull/1200 but was never finalized/merged)
- hard or impossible to "carry over" the cache while moving files around, e.g. by `dandi mv` command: https://github.com/dandi/dandi-cli/issues/848#issuecomment-988055068 and some analysis to make it `blocked` by joblib backend: https://github.com/con/fscacher/issues/57
- can add lots (over 1000%?) of overhead in some cases (e.g., see #914) -- possibly due to needing to create directory with 2 files even for a quick invocation
- wasting filesystem inodes which could be of precious resource on HPC/some filesystems

The question is how we could redesign (or just expand since current implementation is good in its simplicity) fscacher to possibly
- RF `I3`: provide an alternative (to directory with 2 files)
- more efficient in case of working with lots of small files
- less reliant on filesystem and thus more gentle to inodes storage backend
- *might need to have locking mechanisms added to guarantee*
- RF `I2` and/or may be `I1` provide some "locality":
- "dataset level": so we could keep cache within any given dandiset, so if that dandiset is removed, cache goes along with it
- "directory/path" (in some cases only): keep all fingerprints for files within `.zarr/` folder within a singular cache "file" -- would save inodes, for `mv` etc operations would be a matter of copying/changing one such file.
- *in principle, both localities could already be achieved with current fscacher to some degree by establishing separate cache per specific dataset/zarr file. We just would need to get away from using a simple generic `@decorator` form and instantiate cache per each dandiset/zarr file in a given location. For 'zarr' support though some alternative storage backend would be needed*
- gain more versatile cleaning
- may be tie a little with the code, i.e. removal of file by the code could trigger cleaning the cache for that path. although not sure if worth adding such explicit ties really.
- support "early decision" for directories (say that changed - run the target compute while in parallel finishing getting the full directory fingerprint; might need storing "tree" fingerprint at least to some level depth)

edit: adding an explicit section

### Features

- be able to query the status/if changed on a path (directory). Will be of use for DataLad as well. So we might need to abstract those interfaces

WDYT @jwodder -- have any ideas/vision on what next design in fscacher could be to make our life in dandi-cli better?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.