fsspec / fsspec/filesystem_spec
FileNotFound for s3 file with / in the key
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
Hello! I am trying to read a csv stored in a remote s3 bucket with a / in the name, like this: 's3://mybucket/path/7/update//nicktorba.part_00000'
When i run this code, I get FileNotFound error:
(as far as I can tell, when I run a pandas read_csv, this is the code used to read the file)
file_obj = fsspec.open(
filepath, mode="rb", **(storage_options or {})
).open()
However, this code runs successfully:
s3_client = boto3.client('s3', aws_access_key_id=storage_options["key"], aws_secret_access_key=storage_options["secret"])
s3_object = s3_client.get_object(
Bucket="my-bucket",
Key="path/7/update//nicktorba.part_00000"
)
df = pd.read_csv(s3_object['Body'], nrows=5)
Is there any way I can update the args to the fspec.open in the first code snippet to have it successfully read the file?
I'm positive it exists and my access is set up correctly because the second snippet works.
Thank you!
(side note: I am unfortunately not the one in charge of the file naming, so removing the slash isn't an option at the moment)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the supplied s3 URL through fsspec.open with the same storage_options, then compare it with boto3 get_object using the same key. Trace the fsspec.open path-handling entry point and existing S3 tests, if present; the issue is done when the double-slash key can be read through fsspec and a regression check covers it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, pandas, python
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100