fsspec / fsspec/filesystem_spec
RF idea: deprecate `same_names` in favor of a more generic `layout` parameter
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
ATM CachingFileSystem has a single bool option same_names to switch layout of files from /hash to /url-filename and thus does not leave room for "improvement":
Under heavy use of the cache use having a flat tree of files (/hash or /url-filename based) could lead to a very heavy directory so filesystem could become inefficient in listing that directory etc.
- A common (look under
.git/objects, same approach used by git-annex, girder etc) workaround is to establish leading directories, e.g. for a/hashit could be/hash[:2]/hash[2:4]/hash[4:]path to the file, thus reducing impact on the file system - for url-based path, it could simply be a path constructed from URI components, e.g. for
http://domain/p1/p2/filenameURL it could becomehttp/domain/p1/p2/filenamepath, thus allowing to disambiguate between file systems etc, and also avoid conflicts for the same common filename (as I guess would be now withsame_names=True).
With above in mind, I think it would have been nice if instead of same_names there was a layout={hash,hashtree,url_filename,url_fullpath} or alike, thus allowing users to switch to most appropriate layout depending on their use case.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating CachingFileSystem and the same_names option. Review how the current /hash and /url-filename layouts are selected, then compare the proposed hash-tree and URL-full-path layouts against existing cache behavior. Done means an agreed layout parameter design that supports the proposed modes and avoids oversized or conflicting directories.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- operating-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100