fsspec / fsspec/filesystem_spec
DirFileSystem not forwarding 'local_file'
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
My use case: I'm using the following snippet to create a virtual filesystem, and create an OpenFile handle. Important to me is that I have a common base class/interface (OpenFile) to pass on to my "business logic" because I've extended it already with types. fs.open() is returning different things depending on the fs, which by itself would be ok, but in combination with the requirement below, I prefer OpenFile over open().
fs = fsspec.implementations.dirfs.DirFileSystem("./sample_data/", fs=fsspec.filesystem("file"))
of = OpenFile(fs, my_file, mode="rb")
In my business logic I'm either loading the whole file in memory (bytes), or memory map it (large files). For this decision I need to know whether it is a local file (incl. cached locally) or not. Only local files can be mmapped. open_local() is testing the presence of a local_file attribute. Obviously a bit of a hack, and not yet part of a spec or contract. I tried to use the hack with the snippet above, but it doesn't work. DirFileSystem is not initialising a local_file attribute. So, right now I need to test for type DirFileSystem, if yes, then test for the local_file attribute, ...
I'm new to fsspec, so may be I'm missing a trick. I wish fsspec would have a clear API for local files (incl. locally cached files), e.g.:
bool = of.has_local_file() # True, if the file is local or have locally cached version
fn = of.local_file() # local file path. Raise exception if only remote
Both should return valid responses, even if the locally cached file does not yet exist. And they should not trigger a download.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the DirFileSystem, OpenFile, and open_local() behavior described in the issue, focusing on how the local_file attribute is handled. The desired result is a defined way to identify and retrieve a local path without triggering a download, but the API shape and project entry points still need to be established.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100