apache / apache/arrow

A libcurl function was given a bad argument

Open
#35,365 7 comments 2 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

We use pyarrow to read data from S3 and sometimes we get the following error:

```
File "/usr/local/lib/python3.10/dist-packages/{org}/store/storage.py", line 794, in _load_partition
table = ds.dataset(
File "pyarrow/_dataset.pyx", line 369, in pyarrow._dataset.Dataset.to_table
File "pyarrow/_dataset.pyx", line 2818, in pyarrow._dataset.Scanner.to_table
File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 115, in pyarrow.lib.check_status
OSError: AWS Error NETWORK_CONNECTION during GetObject operation: curlCode: 43, A libcurl function was given a bad argument
```

We were trying to find the reason why it happens, but it is very random. Can you help to understand where actually the problem with libcurl can be?

### Component(s)

Python

Contributor guide

Open the contributing guide

Research direction

Start with the _load_partition entry point in storage.py at line 794 and the pyarrow dataset call shown in the traceback. Reproduce the intermittent S3 GetObject failure if possible and trace the curlCode 43 path; done means identifying a reproducible cause and a confirmed fix or clear upstream diagnosis.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.