apache / apache/arrow

Error: AWS Error [code 99]: curlCode: 28, Timeout was reached

Open
#33,371 0 comments 1 reaction 0 assignees View on GitHub
Component: Parquet Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

We have AWS Lambdas that were updated to run with PyArrow 9.  We were calling the pyarrow.parquet.read_table function with the columns parameter to specify which columns to read.  With PyArrow version prior to 9 this worked fine with just 500MB of memory for the Lambda.  With PyArrow 9 we started to get these exceptions:

AWS Error [code 99]: curlCode: 28, Timeout was reached

We did some experimenting and found two options to get around the issue.

Option 1:  Increase the memory to 1.7 MB so the Lamba gets a full CPU for the container
Option 2:  Continue to allocate just 500 MB of memory, but stop using the columns parameter.

 

Looking for some insight as to what the issue might be.  Thanks.

**Reporter**: [John](https://issues.apache.org/jira/browse/ARROW-18182)

**Note**: *This issue was originally created as [ARROW-18182](https://issues.apache.org/jira/browse/ARROW-18182). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the AWS Lambda case around pyarrow.parquet.read_table with PyArrow 9, comparing the columns parameter against reading all columns at 500 MB and higher memory. Check whether the timeout is reproducible and identify the relevant PyArrow or Parquet behavior; done means the cause is isolated and a verified fix or clear limitation is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.