fsspec / fsspec/filesystem_spec

Recover full name for `OpenFile` objects

Open
#1,459 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

Currently fsspec.core.OpenFile objects have a property called full_name which I think should return the full file name i.e. the protocol + local path. I think it would be nice that user should be able to pass this property to fsspec.core.url_to_fs and produce the same filesystem and local path.

The current implementation of this property just prepends the protocol to the local path, which is not enough in some cases where the protocol has additional parameters, such as ssh which contains the host, etc.

The need comes from wanting to expand a glob expression but return the full paths:

files: list[str] = [f.full_name for f in fsspec.open_files("glob_expression")]

Is not enough as stated and you cannot typically open files from these strings as the filesystem lacks necessary parameters.

I tried to find a way to achieve this but I think it's not possible at the moment. I guess the _unstrip_protocol of each filesystem would need to be updated.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with fsspec.core.OpenFile.full_name and the filesystem _unstrip_protocol implementations, then trace how fsspec.core.url_to_fs reconstructs a filesystem and path. Check the behavior through fsspec.open_files("glob_expression") and verify that each returned full_name preserves protocol parameters and can be passed back to url_to_fs to recover the same filesystem and local path.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.