fsspec / fsspec/filesystem_spec

TypeError: ClientSession._request() got an unexpected keyword argument 's3'

Open
#1,925 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

I have a list of URLs that are mixed with different schemes, some in the format s3:// and others in https://. I want to use one global storage option to read them all. However, the HTTP backend is attempting to use the S3 storage option.

Different protocol backends should only use their corresponding protocol storage options.

import fsspec

storage_options = {
    "s3": {
        "key": "My key",
        "secret": "My secret",
        "use_ssl": True,
    },
}

for url in url_list:
    with fsspec.open(url, "rb", **storage_options) as f:
        data = f.read()
        print(data)

error:

    def get(
        self, url: StrOrURL, *, allow_redirects: bool = True, **kwargs: Any
    ) -> "_RequestContextManager":
        """Perform HTTP GET request."""
        return _RequestContextManager(
>           self._request(
                hdrs.METH_GET, url, allow_redirects=allow_redirects, **kwargs
            )
        )
E       TypeError: ClientSession._request() got an unexpected keyword argument 's3'

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the mixed s3:// and https:// URL example and trace how fsspec.open dispatches storage_options to each protocol backend. Confirm that HTTP does not receive the nested S3 options while S3 still receives its own options, then add regression coverage for the mixed-URL case.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.