fsspec / fsspec/filesystem_spec

FileNotFound for s3 file with / in the key

Open
#1,310 11 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

Hello! I am trying to read a csv stored in a remote s3 bucket with a / in the name, like this: 's3://mybucket/path/7/update//nicktorba.part_00000'

When i run this code, I get FileNotFound error:
(as far as I can tell, when I run a pandas read_csv, this is the code used to read the file)

file_obj = fsspec.open(
    filepath, mode="rb", **(storage_options or {})
).open()

However, this code runs successfully:

s3_client = boto3.client('s3', aws_access_key_id=storage_options["key"], aws_secret_access_key=storage_options["secret"])

s3_object = s3_client.get_object(
    Bucket="my-bucket", 
    Key="path/7/update//nicktorba.part_00000"
)

df = pd.read_csv(s3_object['Body'], nrows=5)

Is there any way I can update the args to the fspec.open in the first code snippet to have it successfully read the file?
I'm positive it exists and my access is set up correctly because the second snippet works.

Thank you!

(side note: I am unfortunately not the one in charge of the file naming, so removing the slash isn't an option at the moment)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the supplied s3 URL through fsspec.open with the same storage_options, then compare it with boto3 get_object using the same key. Trace the fsspec.open path-handling entry point and existing S3 tests, if present; the issue is done when the double-slash key can be read through fsspec and a regression check covers it.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, pandas, python
Domain
cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.