fsspec / fsspec/filesystem_spec
LocalFileSystem does not work with CachingFileSystem
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
I am attempting to use the block wise cache to read from networked drive however I am finding that the cache does not store any blocks.
I believe this is due to the fact that LocalFileOpener does not inherit from AbstractBufferedFile but io.IOBase
I have experimented by modifying LocalFileOpener to use AbstractBufferedFile instead and and it appears to work as expected.
Any help would be greatly apprecitated.
Modified LocalFileOpener:
class LocalFileOpener(AbstractBufferedFile):
def __init__(
self, path, mode, autocommit=True, fs=None, compression=None, **kwargs
):
super().__init__(fs, path, mode, **kwargs)
logger.debug("open file: %s", path)
self.f = None
self.compression = get_compression(path, compression)
# self._open()
def _open(self):
if self.f is None or self.f.closed:
if self.autocommit or "w" not in self.mode:
self.f = open(self.path, mode=self.mode)
if self.compression:
compress = compr[self.compression]
self.f = compress(self.f, mode=self.mode)
else:
# TODO: check if path is writable?
i, name = tempfile.mkstemp()
os.close(i) # we want normal open and normal buffered file
self.temp = name
self.f = open(name, mode=self.mode)
if "w" not in self.mode:
self.size = self.f.seek(0, 2)
self.f.seek(0)
self.f.size = self.size
def _fetch_range(self, start, end):
# probably only used by cached FS
if "r" not in self.mode:
raise ValueError
self._open()
self.f.seek(start)
return self.f.read(end - start)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Locate LocalFileOpener, AbstractBufferedFile, and CachingFileSystem, then trace how block-wise cache reads request ranges from local files. Reproduce the networked-drive case and verify that read blocks are stored and reused by the cache without regressing normal local-file and compression behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100