HTTP block download failure with caching when content is gzipped

Start with implementations/http.py, especially async_fetch_range, then trace the cached.py _open and caching.py _fetch calls in the traceback. Reproduce the provided simplecache example with a file larger than the default block size, comparing the second block with Accept-Encoding set to identity. Done means block downloads from the gzipped HTTP response complete without requiring either workaround.

Similar line of investigation as #388, but a distinct issue. I'm running into an issue with the block downloading from an HTTP source when the server tries to serve gzip-encoded data. A minimal example:

import fsspec

path = "simplecache::https://opendata.arcgis.com/datasets/d3cd48afaacd4913b923fd98c6591276_36.zip"

storage_options = {
    "simplecache": {
        "cache_storage": "/tmp/cachetest/",
    }
}

with fsspec.open(path, "rb", **storage_options) as f:
    data = f.read()

produces

Traceback (most recent call last):
  File "test-cache.py", line 16, in <module>
    with fsspec.open(path, "rb", **storage_options) as f:
  File "filesystem_spec/fsspec/core.py", line 101, in __enter__
    f = self.fs.open(self.path, mode=mode)
  File "filesystem_spec/fsspec/implementations/cached.py", line 371, in <lambda>
    return lambda *args, **kw: getattr(type(self), item)(self, *args, **kw)
  File "filesystem_spec/fsspec/spec.py", line 839, in open
    f = self._open(
  File "filesystem_spec/fsspec/implementations/cached.py", line 371, in <lambda>
    return lambda *args, **kw: getattr(type(self), item)(self, *args, **kw)
  File "filesystem_spec/fsspec/implementations/cached.py", line 562, in _open
    data = f.read(f.blocksize)
  File "filesystem_spec/fsspec/implementations/http.py", line 339, in read
    return super().read(length)
  File "filesystem_spec/fsspec/spec.py", line 1310, in read
    out = self.cache._fetch(self.loc, self.loc + length)
  File "filesystem_spec/fsspec/caching.py", line 38, in _fetch
    return self.fetcher(start, stop)
  File "filesystem_spec/fsspec/asyn.py", line 100, in wrapper
    return maybe_sync(func, self, *args, **kwargs)
  File "filesystem_spec/fsspec/asyn.py", line 80, in maybe_sync
    return sync(loop, func, *args, **kwargs)
  File "filesystem_spec/fsspec/asyn.py", line 51, in sync
    raise exc.with_traceback(tb)
  File "filesystem_spec/fsspec/asyn.py", line 35, in f
    result[0] = await future
  File "filesystem_spec/fsspec/implementations/http.py", line 376, in async_fetch_range
    out = await r.read()
  File "lib/python3.8/site-packages/aiohttp/client_reqrep.py", line 973, in read
    self._body = await self.content.read()
  File "lib/python3.8/site-packages/aiohttp/streams.py", line 334, in read
    raise self._exception
aiohttp.client_exceptions.ClientPayloadError: 400, message='Can not decode content-encoding: gzip'

Note that the file itself is not gzipped, this is in the HTTP transport layer.

Weirdly enough, this seems to happen upon downloading the second block of data, the first gets decoded just fine, so this only appears for files that are larger than the default block size. I'm not sure what the cause of that is just yet.

I can fix the above issue by either disabling the blocking (http={"block_size": 0}), or by requesting no gzip-encoding from the server (http={ "headers": { "Accept-Encoding": "identity" } }), but both of these are lower-level fixes than I'd like to point users at.