fsspec / fsspec/filesystem_spec

Missing kwargs in ZipFileSystem leading to `botocore.exceptions.NoCredentialsError: Unable to locate credentials`

Open
#1,573 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

Hello,

Summary

I initially created an issue on the xarray repository: https://github.com/pydata/xarray/issues/8944 ; I place it here for reference as it provides more context.

The part concerning fsspec is the following: when trying to access a Zip file on a s3 bucket requiring authentication, the kwargs are not passed in an open method called by ZipFileSystem.__init__, leading to an authentication error. The problematic line is: https://github.com/fsspec/filesystem_spec/blob/37c1bc63b9c5a5b2b9a0d5161e89b4233f888b29/fsspec/implementations/zip.py#L57

Version: fsspec: 2023.10.0 (also true for current version mentioned above): kwargs are not passed to the open method in .

Suggested bugfix

Current:

        fo = fsspec.open(
            fo, mode=mode + "b", protocol=target_protocol, **(target_options or {})
        )

Proposed bugfix: (passing the kwargs)

        fo = fsspec.open(
            fo, mode=mode + "b", protocol=target_protocol, **(target_options or {}), **kwargs
        )

When testing locally, adding the kwargs leads to a successful opening of the Zip file. I don't have an Minimal Complete Verifiable unfortunately since it requires a Zip file on an s3 bucket with authentication.

Do you think this would be a good solution, or maybe something more suble might be needed?

Thanks!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in fsspec/implementations/zip.py at the ZipFileSystem.init call around line 57, and trace how options reach fsspec.open. Verify the proposed argument flow against an authenticated S3 ZIP access; done means the archive opens without botocore.exceptions.NoCredentialsError.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.