fsspec / fsspec/filesystem_spec
Pass runtime parameters only to specific implementation
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
Config file based options for passing specific arguments to specific backends don't work for non-string parameters.
For example, from the wording of #453 if I want to pass a custom google.auth.Credentials object to gcs then
I expected be able to do one of the following:
file.open(path, gcs_token=creds)
file.open(path, gcs={'token': creds})
But neither work (evident why from looking at the code, there's actually no change to how runtime parameters are parsed, only config files). There doesn't appear to be any way to force runtime parameters that can't be serialized into a json or ini file to be passed to a specific backend.
The only available workaround is something like:
fs, _ = fsspec.url_to_fs(path)
storage_options = {}
if isinstance(fs, gcsfs.GCSFileSystem):
storage_options['token'] = creds
return fsspec.open(path, **storage_options)
Essentially building the storage class twice and requiring an explicit check. Eliminating a lot of the utility of fsspec in the first place.
I understand that implementations may have keyword arguments that clash with protocol names, hence not wanting to use the first solution. But it seems reasonable to have another argument, "protocol_defaults" which allows passing a config dictionary directly.
fsspec.open(path, protocol_defaults={'gcs': {'token': creds}})
This could be implemented internally with a context manager (also useful for other functions):
with fsspec.config.protocol_defaults(gcs={'token': creds}):
fs, _ = fsspec.url_to_fs(path)
fs.get(path, local_path)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with fsspec.open and fsspec.url_to_fs, then review how runtime parameters and config-file options are parsed. Determine how a non-serializable value such as a gcs token can be scoped to one backend without clashing with protocol names. Done means calls using protocol_defaults or an equivalent mechanism pass those values only to the selected backend.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100