fsspec / fsspec/filesystem_spec

v 2021.4.0 causes ValueError's in HTTP calls

Open
#620 14 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

Since fsspec is used in intake/intake-catalog the latest version is causing issues reading from an HTTP Catalog

"/Users/donghunl/miniconda3/envs/intake_dal37/lib/python3.7/site-packages/intake_dal/dal_catalog.py", line 69, in __init__
    super(DalCatalog, self).__init__(path, autoreload, **kwargs)
  File "/Users/donghunl/miniconda3/envs/intake_dal37/lib/python3.7/site-packages/intake_nested_yaml_catalog/nested_yaml_catalog.py", line 75, in __init__
    super(NestedYAMLFileCatalog, self).__init__(path, autoreload, **kwargs)
  File "/Users/donghunl/miniconda3/envs/intake_dal37/lib/python3.7/site-packages/intake/catalog/local.py", line 565, in __init__
    super(YAMLFileCatalog, self).__init__(**kwargs)
  File "/Users/donghunl/miniconda3/envs/intake_dal37/lib/python3.7/site-packages/intake/catalog/base.py", line 113, in __init__
    self.force_reload()
  File "/Users/donghunl/miniconda3/envs/intake_dal37/lib/python3.7/site-packages/intake/catalog/base.py", line 172, in force_reload
    self._load()
  File "/Users/donghunl/miniconda3/envs/intake_dal37/lib/python3.7/site-packages/intake_dal/dal_catalog.py", line 87, in _load
    super()._load()
  File "/Users/donghunl/miniconda3/envs/intake_dal37/lib/python3.7/site-packages/intake/catalog/local.py", line 591, in _load
    text = f.read().decode()
  File "/Users/donghunl/miniconda3/envs/intake_dal37/lib/python3.7/site-packages/fsspec/implementations/http.py", line 457, in read
    return super().read(length)
  File "/Users/donghunl/miniconda3/envs/intake_dal37/lib/python3.7/site-packages/fsspec/spec.py", line 1449, in read
    out = self.cache._fetch(self.loc, self.loc + length)
  File "/Users/donghunl/miniconda3/envs/intake_dal37/lib/python3.7/site-packages/fsspec/caching.py", line 376, in _fetch
    self.cache = self.fetcher(start, bend)
  File "/Users/donghunl/miniconda3/envs/intake_dal37/lib/python3.7/site-packages/fsspec/asyn.py", line 72, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/Users/donghunl/miniconda3/envs/intake_dal37/lib/python3.7/site-packages/fsspec/asyn.py", line 53, in sync
    raise result[0]
  File "/Users/donghunl/miniconda3/envs/intake_dal37/lib/python3.7/site-packages/fsspec/asyn.py", line 20, in _runner
    result[0] = await coro
  File "/Users/donghunl/miniconda3/envs/intake_dal37/lib/python3.7/site-packages/fsspec/implementations/http.py", line 507, in async_fetch_range
    "Got more bytes (%i) than requested (%i)" % (cl, end - start)
ValueError: Got more bytes (117554) than requested (31)

Downgrading to 0.8.3 resolves the issue

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce loading the HTTP Catalog with the affected fsspec version, starting from implementations/http.py at async_fetch_range and caching.py at _fetch, as shown in the traceback. Compare the behavior with version 0.8.3; done means HTTP catalog reads no longer raise the reported ValueError without requiring the downgrade.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
networking
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.