fsspec / fsspec/filesystem_spec
Inconsistent use of protocol specific options
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
Setting protocol specific options has been a convenient method for overriding the default options for each protocol. E.g., the Azure blob storage implementation behaves peculiarly and requires setting anon=False to use the credentials in the environment (https://github.com/fsspec/adlfs/issues/348).
So for paths provided by an application, we might do:
fsspec.open(..., az={"anon": False})
This option is ignored for local paths, and used for az:// protocol urls, and therefore allows us to configure defaults for each protocol. Unfortunately, this doesn't work with https(s) protocol urls, since the kwargs are directly forwarded to aiohttp, e.g. https://github.com/fsspec/filesystem_spec/blob/561428ca18a9865d8f63fe188a590d791ec52c92/fsspec/implementations/http.py#L826
- If this is an intended usage mode: How about dropping all protocol specific kwargs before forwarding to the http implementation?
- If this is not an intended feature: How can we set per protocol defaults otherwise? If we have to manually parse the URLs and assign different kwargs to the
fsspec.openmethod the convinience of this API greatly diminishes.
Either way, I'm happy to contribute code if we can agree on a solution.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with fsspec.open and the HTTP implementation at fsspec/implementations/http.py around the referenced line, then trace how protocol-specific options reach aiohttp. Confirm whether dropping those options for HTTP is the intended behavior or determine the alternative for per-protocol defaults; done requires an agreed solution and corresponding tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100