fsspec / fsspec/filesystem_spec

FSSpec's get_mapper and other functions can produce nameless files for S3 Filesystems

Open
#1,554 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

Observed Behavior:

When fsspec's S3FS's get_mapper function is called on an empty directory-like path in an S3 filesystem, it can produce a dictionary-like object that is non-empty, and includes the key '' (i.e. the empty string), and can produce files that are in fact empty directories in its listing:

image

It appears similar behavior can happen with other functions like ls or walk.

Expected Behavior
  1. get_mapper should not produce empty string filenames in directories, as users have to filter these out (they appear not to actually exist, and may correspond to versioning artifacts?).
  2. get_mapper should not produce keys that correspond to empty directories.

Both of these behaviors can currently lead to unusual interactions when using get_mapper. For example, when iterating over an FSMap and copying files from the remote system, trying to copy the '' file will produce an error.

Possible Fix

get_mapper for S3FS could include a filter that avoids adding leaf paths that are not files, tested using the isfile predicate.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the behavior with S3FS's get_mapper on an empty directory-like path, then inspect the related ls and walk behavior and the proposed isfile predicate. Done means listings and mappings omit the empty-string key and entries representing empty directories, with regression coverage for the reported cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.