Error when using with pd.read_parquet with threading on

Start with the reproduction in print_parquet_metadata.py and compare the two pd.read_parquet loops using use_threads=False and use_threads=True. Review the fsspec and s3fs debug log ranges alongside the ArrowIOError, then verify the fix by repeatedly reading the same S3 Parquet file with threading enabled without intermittent truncated-page errors.

Since updating to latest version I randomly get errors like this:

Traceback (most recent call last):
  File "print_parquet_metadata.py", line 28, in <module>
    df = pd.read_parquet(args.file, use_threads=True)
  File "/Users/mtrbean/.pyenv/versions/stats3/lib/python3.6/site-packages/pandas/io/parquet.py", line 282, in read_parquet
    return impl.read(path, columns=columns, **kwargs)
  File "/Users/mtrbean/.pyenv/versions/stats3/lib/python3.6/site-packages/pandas/io/parquet.py", line 129, in read
    **kwargs).to_pandas()
  File "/Users/mtrbean/.pyenv/versions/stats3/lib/python3.6/site-packages/pyarrow/parquet.py", line 1216, in read_table
    use_pandas_metadata=use_pandas_metadata)
  File "/Users/mtrbean/.pyenv/versions/stats3/lib/python3.6/site-packages/pyarrow/parquet.py", line 216, in read
    use_threads=use_threads)
  File "pyarrow/_parquet.pyx", line 1086, in pyarrow._parquet.ParquetReader.read_all
  File "pyarrow/error.pxi", line 87, in pyarrow.lib.check_status
pyarrow.lib.ArrowIOError: Unexpected end of stream: Page was smaller (26292) than expected (59582)

I'm pretty sure that the file is not corrupted because sometimes it can be read successfully, and it can also be read when the file is local.

It is also very hard to reproduce consistently but if I try to read the file in a loop, I realize a pattern:

This will succeed 30 times in a row without a problem

f = "s3://<bucket>/<parquet_file>"
for i in range(30):
    pd.read_parquet(f, use_threads=False)

This will fail sometimes on the first iteration, sometimes on the second, and never got to iteration number 5:

f = "s3://<bucket>/<parquet_file>"
for i in range(30):
    pd.read_parquet(f, use_threads=True)

I suspect that it has something to do with random access with multiple threads, and the cache is not handling it correctly.

Finally this is the debug log which I hope will help

DEBUG:urllib3.util.retry:Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): mybucket.s3.us-west-2.amazonaws.com
DEBUG:urllib3.connectionpool:https://mybucket.s3.us-west-2.amazonaws.com:443 "GET /?list-type=2&prefix=&delimiter=%2F&encoding-type=url HTTP/1.1" 200 None
DEBUG:urllib3.util.retry:Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
DEBUG:urllib3.connectionpool:https://mybucket.s3.us-west-2.amazonaws.com:443 "GET /?list-type=2&prefix=temp%2F&delimiter=%2F&encoding-type=url HTTP/1.1" 200 None
DEBUG:fsspec:<File-like object S3FileSystem, mybucket/temp/data.parquet> read: 18606060 - 18671596
DEBUG:s3fs.core:Fetch: mybucket/temp/data.parquet, 18606060-23914476
DEBUG:urllib3.util.retry:Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
DEBUG:urllib3.connectionpool:https://mybucket.s3.us-west-2.amazonaws.com:443 "GET /temp/data.parquet HTTP/1.1" 206 65536
DEBUG:fsspec:<File-like object S3FileSystem, mybucket/temp/data.parquet> read: 18176581 - 18670182
DEBUG:s3fs.core:Fetch: mybucket/temp/data.parquet, 18176581-18606060
DEBUG:urllib3.util.retry:Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
DEBUG:urllib3.connectionpool:https://mybucket.s3.us-west-2.amazonaws.com:443 "GET /temp/data.parquet HTTP/1.1" 206 429479
DEBUG:fsspec:<File-like object S3FileSystem, mybucket/temp/data.parquet> read: 4 - 7237571
DEBUG:s3fs.core:Fetch: mybucket/temp/data.parquet, 4-12480451
DEBUG:urllib3.util.retry:Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
DEBUG:urllib3.connectionpool:https://mybucket.s3.us-west-2.amazonaws.com:443 "GET /temp/data.parquet HTTP/1.1" 206 12480447
DEBUG:fsspec:<File-like object S3FileSystem, mybucket/temp/data.parquet> read: 8274755 - 18176458
DEBUG:s3fs.core:Fetch: mybucket/temp/data.parquet, 8274755-10485764
DEBUG:urllib3.util.retry:Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
DEBUG:urllib3.connectionpool:https://mybucket.s3.us-west-2.amazonaws.com:443 "GET /temp/data.parquet HTTP/1.1" 206 2211009
DEBUG:fsspec:<File-like object S3FileSystem, mybucket/temp/data.parquet> read: 7237648 - 8274676
DEBUG:s3fs.core:Fetch: mybucket/temp/data.parquet, 7237648-8274755
DEBUG:urllib3.util.retry:Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
DEBUG:urllib3.connectionpool:https://mybucket.s3.us-west-2.amazonaws.com:443 "GET /temp/data.parquet HTTP/1.1" 206 1037107
Traceback (most recent call last):
  File "print_parquet_metadata.py", line 28, in <module>
    df = pd.read_parquet(args.file, use_threads=True)
  File "/Users/mtrbean/.pyenv/versions/stats3/lib/python3.6/site-packages/pandas/io/parquet.py", line 282, in read_parquet
    return impl.read(path, columns=columns, **kwargs)
  File "/Users/mtrbean/.pyenv/versions/stats3/lib/python3.6/site-packages/pandas/io/parquet.py", line 129, in read
    **kwargs).to_pandas()
  File "/Users/mtrbean/.pyenv/versions/stats3/lib/python3.6/site-packages/pyarrow/parquet.py", line 1216, in read_table
    use_pandas_metadata=use_pandas_metadata)
  File "/Users/mtrbean/.pyenv/versions/stats3/lib/python3.6/site-packages/pyarrow/parquet.py", line 216, in read
    use_threads=use_threads)
  File "pyarrow/_parquet.pyx", line 1086, in pyarrow._parquet.ParquetReader.read_all
  File "pyarrow/error.pxi", line 87, in pyarrow.lib.check_status
pyarrow.lib.ArrowIOError: Unexpected end of stream: Page was smaller (26292) than expected (59582)