fsspec / fsspec/filesystem_spec

Storage options passed to unexpected backend

Open
#388 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

I'm looking into chaining caching with HTTP/S3 access as part of intake/intake_geopandas#15, and found some unexpected behavior with with chaining. My expectation of **storage_options was that each dict would be passed as kwargs to the relevant sub-filesystem. The problem I ran into is this: suppose I have a set of paths, some of which are intended to be cached and some of which are not. In that case, the HTTPFileSystem without caching was passed the caching options, and it passed those options as kwargs to requests, which failed:

import fsspec

path = "https://github.com/intake/filesystem_spec/raw/master/README.md"

# Succeeds
with fsspec.open("simplecache::" + path, simplecache={"cache_storage": "/tmp/testcache"}) as f:
    print(f.read())

# Fails as simplecache is passed to requests
with fsspec.open(path, simplecache={"cache_storage": "/tmp/testcache"}) as f:
    print(f.read())

I suppose I had expected storage options for filesystems not involved in a given path to be ignored. Is this behavior expected?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the two fsspec.open examples, then trace how storage_options are routed through the simplecache and HTTPFileSystem backends. Check the option handling around requests and filesystem chaining; done means options for backends not used by a path are ignored without breaking the existing cached-path behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.