fsspec / fsspec/filesystem_spec
Missing kwargs in ZipFileSystem leading to `botocore.exceptions.NoCredentialsError: Unable to locate credentials`
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
Hello,
Summary
I initially created an issue on the xarray repository: https://github.com/pydata/xarray/issues/8944 ; I place it here for reference as it provides more context.
The part concerning fsspec is the following: when trying to access a Zip file on a s3 bucket requiring authentication, the kwargs are not passed in an open method called by ZipFileSystem.__init__, leading to an authentication error. The problematic line is: https://github.com/fsspec/filesystem_spec/blob/37c1bc63b9c5a5b2b9a0d5161e89b4233f888b29/fsspec/implementations/zip.py#L57
Version: fsspec: 2023.10.0 (also true for current version mentioned above): kwargs are not passed to the open method in .
Suggested bugfix
Current:
fo = fsspec.open(
fo, mode=mode + "b", protocol=target_protocol, **(target_options or {})
)
Proposed bugfix: (passing the kwargs)
fo = fsspec.open(
fo, mode=mode + "b", protocol=target_protocol, **(target_options or {}), **kwargs
)
When testing locally, adding the kwargs leads to a successful opening of the Zip file. I don't have an Minimal Complete Verifiable unfortunately since it requires a Zip file on an s3 bucket with authentication.
Do you think this would be a good solution, or maybe something more suble might be needed?
Thanks!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in fsspec/implementations/zip.py at the ZipFileSystem.init call around line 57, and trace how options reach fsspec.open. Verify the proposed argument flow against an authenticated S3 ZIP access; done means the archive opens without botocore.exceptions.NoCredentialsError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100