fsspec / fsspec/filesystem_spec
RF idea: sparse-file cache information to be stored per-cached file (instead of a singular `/cache`)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
Related to #555 (as if the layout to change) and may be other issues (e.g. #553 ) on closing/reopening cache. I could be totally wrong, but ATM I think fsspec stores cache information (e.g. which blocks of the sparse file were already fetched etc?) in a singular /cache file within the cache directory. I wonder, if e.g. cache is to store thousands of files - how efficient dumping of the cache information would be?
Ideally cache meta data should be updated "on disk" as soon as a new block for a file is fetched/stored into the cache sparse file, so there would be no need for any additional metadata dump upon close or __del__, but likely it would be an expensive operation if it is the entire dump of metadata for all files to be dumped at once.
So I wonder if it is worth adopting the same approach as rclone took for its mount operation. It has two directories in the cache -- vfs and vfsMeta where in vfs are the actual sparse files, and in vfsMeta (I guess) under the same key path, all the information about that particular cached file, e.g.
$> ls -ld /home/yoh/.cache/rclone/vfs*/s3_us_east_2/dandiarchive/largeuploads/blobs/f5d/aac/79d9c6c1ba2439200d45680f1cb941087750a000e484ea2973232709d2
-rw------- 1 yoh yoh 3628888939868 Feb 18 20:57 /home/yoh/.cache/rclone/vfs/s3_us_east_2/dandiarchive/largeuploads/blobs/f5d/aac/79d9c6c1ba2439200d45680f1cb941087750a000e484ea2973232709d2
-rw------- 1 yoh yoh 261 Mar 3 11:14 /home/yoh/.cache/rclone/vfsMeta/s3_us_east_2/dandiarchive/largeuploads/blobs/f5d/aac/79d9c6c1ba2439200d45680f1cb941087750a000e484ea2973232709d2
such layout also makes it easy with regular rm to prune cache for no longer desired files etc.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start by tracing how fsspec currently stores and reloads sparse-file cache metadata, then compare the related issues #555 and #553. Done would require an agreed per-file metadata layout and a clear update and close/reopen lifecycle for large caches.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100