Error: AWS Error [code 99]: curlCode: 28, Timeout was reached
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
We have AWS Lambdas that were updated to run with PyArrow 9. We were calling the pyarrow.parquet.read_table function with the columns parameter to specify which columns to read. With PyArrow version prior to 9 this worked fine with just 500MB of memory for the Lambda. With PyArrow 9 we started to get these exceptions:
AWS Error [code 99]: curlCode: 28, Timeout was reached
We did some experimenting and found two options to get around the issue.
Option 1: Increase the memory to 1.7 MB so the Lamba gets a full CPU for the container
Option 2: Continue to allocate just 500 MB of memory, but stop using the columns parameter.
Looking for some insight as to what the issue might be. Thanks.
**Reporter**: [John](https://issues.apache.org/jira/browse/ARROW-18182)
**Note**: *This issue was originally created as [ARROW-18182](https://issues.apache.org/jira/browse/ARROW-18182). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start by reproducing the AWS Lambda case around pyarrow.parquet.read_table with PyArrow 9, comparing the columns parameter against reading all columns at 500 MB and higher memory. Check whether the timeout is reproducible and identify the relevant PyArrow or Parquet behavior; done means the cause is isolated and a verified fix or clear limitation is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100