fsspec / fsspec/filesystem_spec
v 2021.4.0 causes ValueError's in HTTP calls
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
Since fsspec is used in intake/intake-catalog the latest version is causing issues reading from an HTTP Catalog
"/Users/donghunl/miniconda3/envs/intake_dal37/lib/python3.7/site-packages/intake_dal/dal_catalog.py", line 69, in __init__
super(DalCatalog, self).__init__(path, autoreload, **kwargs)
File "/Users/donghunl/miniconda3/envs/intake_dal37/lib/python3.7/site-packages/intake_nested_yaml_catalog/nested_yaml_catalog.py", line 75, in __init__
super(NestedYAMLFileCatalog, self).__init__(path, autoreload, **kwargs)
File "/Users/donghunl/miniconda3/envs/intake_dal37/lib/python3.7/site-packages/intake/catalog/local.py", line 565, in __init__
super(YAMLFileCatalog, self).__init__(**kwargs)
File "/Users/donghunl/miniconda3/envs/intake_dal37/lib/python3.7/site-packages/intake/catalog/base.py", line 113, in __init__
self.force_reload()
File "/Users/donghunl/miniconda3/envs/intake_dal37/lib/python3.7/site-packages/intake/catalog/base.py", line 172, in force_reload
self._load()
File "/Users/donghunl/miniconda3/envs/intake_dal37/lib/python3.7/site-packages/intake_dal/dal_catalog.py", line 87, in _load
super()._load()
File "/Users/donghunl/miniconda3/envs/intake_dal37/lib/python3.7/site-packages/intake/catalog/local.py", line 591, in _load
text = f.read().decode()
File "/Users/donghunl/miniconda3/envs/intake_dal37/lib/python3.7/site-packages/fsspec/implementations/http.py", line 457, in read
return super().read(length)
File "/Users/donghunl/miniconda3/envs/intake_dal37/lib/python3.7/site-packages/fsspec/spec.py", line 1449, in read
out = self.cache._fetch(self.loc, self.loc + length)
File "/Users/donghunl/miniconda3/envs/intake_dal37/lib/python3.7/site-packages/fsspec/caching.py", line 376, in _fetch
self.cache = self.fetcher(start, bend)
File "/Users/donghunl/miniconda3/envs/intake_dal37/lib/python3.7/site-packages/fsspec/asyn.py", line 72, in wrapper
return sync(self.loop, func, *args, **kwargs)
File "/Users/donghunl/miniconda3/envs/intake_dal37/lib/python3.7/site-packages/fsspec/asyn.py", line 53, in sync
raise result[0]
File "/Users/donghunl/miniconda3/envs/intake_dal37/lib/python3.7/site-packages/fsspec/asyn.py", line 20, in _runner
result[0] = await coro
File "/Users/donghunl/miniconda3/envs/intake_dal37/lib/python3.7/site-packages/fsspec/implementations/http.py", line 507, in async_fetch_range
"Got more bytes (%i) than requested (%i)" % (cl, end - start)
ValueError: Got more bytes (117554) than requested (31)
Downgrading to 0.8.3 resolves the issue
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce loading the HTTP Catalog with the affected fsspec version, starting from implementations/http.py at async_fetch_range and caching.py at _fetch, as shown in the traceback. Compare the behavior with version 0.8.3; done means HTTP catalog reads no longer raise the reported ValueError without requiring the downgrade.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100