fsspec / fsspec/filesystem_spec
Storage options passed to unexpected backend
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
I'm looking into chaining caching with HTTP/S3 access as part of intake/intake_geopandas#15, and found some unexpected behavior with with chaining. My expectation of **storage_options was that each dict would be passed as kwargs to the relevant sub-filesystem. The problem I ran into is this: suppose I have a set of paths, some of which are intended to be cached and some of which are not. In that case, the HTTPFileSystem without caching was passed the caching options, and it passed those options as kwargs to requests, which failed:
import fsspec
path = "https://github.com/intake/filesystem_spec/raw/master/README.md"
# Succeeds
with fsspec.open("simplecache::" + path, simplecache={"cache_storage": "/tmp/testcache"}) as f:
print(f.read())
# Fails as simplecache is passed to requests
with fsspec.open(path, simplecache={"cache_storage": "/tmp/testcache"}) as f:
print(f.read())
I suppose I had expected storage options for filesystems not involved in a given path to be ignored. Is this behavior expected?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the two fsspec.open examples, then trace how storage_options are routed through the simplecache and HTTPFileSystem backends. Check the option handling around requests and filesystem chaining; done means options for backends not used by a path are ignored without breaking the existing cached-path behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100