fsspec / fsspec/filesystem_spec
TypeError: ClientSession._request() got an unexpected keyword argument 's3'
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
I have a list of URLs that are mixed with different schemes, some in the format s3:// and others in https://. I want to use one global storage option to read them all. However, the HTTP backend is attempting to use the S3 storage option.
Different protocol backends should only use their corresponding protocol storage options.
import fsspec
storage_options = {
"s3": {
"key": "My key",
"secret": "My secret",
"use_ssl": True,
},
}
for url in url_list:
with fsspec.open(url, "rb", **storage_options) as f:
data = f.read()
print(data)
error:
def get(
self, url: StrOrURL, *, allow_redirects: bool = True, **kwargs: Any
) -> "_RequestContextManager":
"""Perform HTTP GET request."""
return _RequestContextManager(
> self._request(
hdrs.METH_GET, url, allow_redirects=allow_redirects, **kwargs
)
)
E TypeError: ClientSession._request() got an unexpected keyword argument 's3'
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the mixed s3:// and https:// URL example and trace how fsspec.open dispatches storage_options to each protocol backend. Confirm that HTTP does not receive the nested S3 options while S3 still receives its own options, then add regression coverage for the mixed-URL case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100