fsspec / fsspec/filesystem_spec

RF idea: sparse-file cache information to be stored per-cached file (instead of a singular `/cache`)

Open
#556 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

Related to #555 (as if the layout to change) and may be other issues (e.g. #553 ) on closing/reopening cache. I could be totally wrong, but ATM I think fsspec stores cache information (e.g. which blocks of the sparse file were already fetched etc?) in a singular /cache file within the cache directory. I wonder, if e.g. cache is to store thousands of files - how efficient dumping of the cache information would be?
Ideally cache meta data should be updated "on disk" as soon as a new block for a file is fetched/stored into the cache sparse file, so there would be no need for any additional metadata dump upon close or __del__, but likely it would be an expensive operation if it is the entire dump of metadata for all files to be dumped at once.

So I wonder if it is worth adopting the same approach as rclone took for its mount operation. It has two directories in the cache -- vfs and vfsMeta where in vfs are the actual sparse files, and in vfsMeta (I guess) under the same key path, all the information about that particular cached file, e.g.

$> ls -ld /home/yoh/.cache/rclone/vfs*/s3_us_east_2/dandiarchive/largeuploads/blobs/f5d/aac/79d9c6c1ba2439200d45680f1cb941087750a000e484ea2973232709d2 
-rw------- 1 yoh yoh 3628888939868 Feb 18 20:57 /home/yoh/.cache/rclone/vfs/s3_us_east_2/dandiarchive/largeuploads/blobs/f5d/aac/79d9c6c1ba2439200d45680f1cb941087750a000e484ea2973232709d2
-rw------- 1 yoh yoh           261 Mar  3 11:14 /home/yoh/.cache/rclone/vfsMeta/s3_us_east_2/dandiarchive/largeuploads/blobs/f5d/aac/79d9c6c1ba2439200d45680f1cb941087750a000e484ea2973232709d2

such layout also makes it easy with regular rm to prune cache for no longer desired files etc.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files, tests, or entry points are named. Start by tracing how fsspec currently stores and reloads sparse-file cache metadata, then compare the related issues #555 and #553. Done would require an agreed per-file metadata layout and a clear update and close/reopen lifecycle for large caches.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.