fsspec / fsspec/filesystem_spec

Feature Request: single environment variable config for protocol

Open
#1,130 6 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

Currently it is possible to provide overrides with environment variables of the style FSSPEC_{protocol}_{kwargname}=value. However, this is limited since it is not possible to provide a dict value. One example where this is needed is s3 with a custom endpoint_url, see https://github.com/fsspec/s3fs/issues/432.

I propose to support a single environment variable FSSPEC_{protocol} expecting its value to be json or yaml (parsed with pyyaml). It is just an extension so there would be no breaking changes. Current individual variables FSSPEC_{protocol}_{kwargname}=value would override the more general FSSPEC_{protocol}.

For example for the s3 protocol, the environment variable name would be FSSPEC_S3. Someone could set shell variables as:

$ FSSPEC_S3='
{
  "key": "my_key",
  "secret": "my_secret",
  "client_kwargs": {
    "endpoint_url": "https://custom-s3-host"
  }
}'
$ FSSPEC_S3_SECRET='my_other_secret'

Then the final arguments for the s3 protocol would be:

key = 'my_key'
secret = 'my_other_secret'
client_kwargs = {'endpoint_url': 'https://custom-s3-host'}

The only possible way to set endpoint_url would be through FSSPEC_S3, since for a more specific variable name such as FSSPEC_S3_CLIENT_KWARGS__ENDPOINT_URL there is no way to distinguish between an item child of a dict in client_kwargs or an argument whose name is client_kwargs__endpoint_url. This is not a limitation since endpoint_url without any other arguments can be set as:

$ FSSPEC_S3='
{
  "client_kwargs": {
    "endpoint_url": "https://custom-s3-host"
  }
}'

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Locate the environment-variable override handling in the filesystem_spec codebase and inspect how protocol kwargs are currently parsed and merged. Implement the documented FSSPEC_{protocol} JSON or YAML input with specific variables taking precedence, then add or update tests covering nested values and override ordering.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, yaml
Domain
tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.