apache / apache/arrow

[C++] Reading file from S3

Open
#41,310 1 comment 0 reactions 1 assignee Claimed by @gitfy View on GitHub
Component: C++ Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

We use arrow s3 filesystem object to read parquet files from object store.

We ran into exceptions like
```
File "/opt/app-root/lib64/python3.9/site-packages/pyarrow/parquet/core.py", line 3008, in read_table
return dataset.read(columns=columns, use_threads=use_threads,
File "/opt/app-root/lib64/python3.9/site-packages/pyarrow/parquet/core.py", line 2636, in read
table = self._dataset.to_table(
File "pyarrow/_dataset.pyx", line 556, in pyarrow._dataset.Dataset.to_table
File "pyarrow/_dataset.pyx", line 3713, in pyarrow._dataset.Scanner.to_table
File "pyarrow/error.pxi", line 154, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status
OSError: Unexpected end of stream: Page was smaller (44336) than expected (44387)
```

and
```
File "pyarrow/error.pxi", line 154, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status
OSError: Couldn't deserialize thrift: don't know what type:
Deserializing page header failed.
```
While investigating this issue, this happens in our system when the file being read is also being overwritten by other processes. There are lots of others who had reported exceptions like above in the past, but most of them got diverted due to the fact they all were using diff filesystem object to read the file. Here we are using the s3fs.cc filesystem from arrow and still had the issues.

Upon reviewing code, while reading a file the following happens
1. Reads the metadata using HEAD request (it gets the content-length)
2. Uses the content length and using GET request with range request to fetch the bytes from object store.

Here it can happen, after the HEAD call, the file might have been updated and while doing the GET the content length can be totally off.

This leads to the above exceptions. I am submitted a minor fix which does the following
1. From the head call, store the version-id of the file along with the content-length
2. Using the version id in the GET call to read the version of the file.
This will avoid running into the race condition, as a new update will create a new file version.

For buckets, which is not version, the only option would be to retry it.

### Component(s)

C++

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.