fsspec / fsspec/filesystem_spec

Inconsistent use of protocol specific options

Open
#1,192 3 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

Setting protocol specific options has been a convenient method for overriding the default options for each protocol. E.g., the Azure blob storage implementation behaves peculiarly and requires setting anon=False to use the credentials in the environment (https://github.com/fsspec/adlfs/issues/348).

So for paths provided by an application, we might do:

fsspec.open(..., az={"anon": False})

This option is ignored for local paths, and used for az:// protocol urls, and therefore allows us to configure defaults for each protocol. Unfortunately, this doesn't work with https(s) protocol urls, since the kwargs are directly forwarded to aiohttp, e.g. https://github.com/fsspec/filesystem_spec/blob/561428ca18a9865d8f63fe188a590d791ec52c92/fsspec/implementations/http.py#L826

  • If this is an intended usage mode: How about dropping all protocol specific kwargs before forwarding to the http implementation?
  • If this is not an intended feature: How can we set per protocol defaults otherwise? If we have to manually parse the URLs and assign different kwargs to the fsspec.open method the convinience of this API greatly diminishes.

Either way, I'm happy to contribute code if we can agree on a solution.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with fsspec.open and the HTTP implementation at fsspec/implementations/http.py around the referenced line, then trace how protocol-specific options reach aiohttp. Confirm whether dropping those options for HTTP is the intended behavior or determine the alternative for per-protocol defaults; done requires an agreed solution and corresponding tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.