fsspec / fsspec/filesystem_spec

DirFileSystem not forwarding 'local_file'

Open
#1,110 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

My use case: I'm using the following snippet to create a virtual filesystem, and create an OpenFile handle. Important to me is that I have a common base class/interface (OpenFile) to pass on to my "business logic" because I've extended it already with types. fs.open() is returning different things depending on the fs, which by itself would be ok, but in combination with the requirement below, I prefer OpenFile over open().

fs = fsspec.implementations.dirfs.DirFileSystem("./sample_data/", fs=fsspec.filesystem("file"))
of = OpenFile(fs, my_file, mode="rb")

In my business logic I'm either loading the whole file in memory (bytes), or memory map it (large files). For this decision I need to know whether it is a local file (incl. cached locally) or not. Only local files can be mmapped. open_local() is testing the presence of a local_file attribute. Obviously a bit of a hack, and not yet part of a spec or contract. I tried to use the hack with the snippet above, but it doesn't work. DirFileSystem is not initialising a local_file attribute. So, right now I need to test for type DirFileSystem, if yes, then test for the local_file attribute, ...

I'm new to fsspec, so may be I'm missing a trick. I wish fsspec would have a clear API for local files (incl. locally cached files), e.g.:

   bool = of.has_local_file()   # True, if the file is local or have locally cached version
   fn = of.local_file()      # local file path. Raise exception if only remote

Both should return valid responses, even if the locally cached file does not yet exist. And they should not trigger a download.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the DirFileSystem, OpenFile, and open_local() behavior described in the issue, focusing on how the local_file attribute is handled. The desired result is a defined way to identify and retrieve a local path without triggering a download, but the API shape and project entry points still need to be established.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.