A libcurl function was given a bad argument
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
We use pyarrow to read data from S3 and sometimes we get the following error:
```
File "/usr/local/lib/python3.10/dist-packages/{org}/store/storage.py", line 794, in _load_partition
table = ds.dataset(
File "pyarrow/_dataset.pyx", line 369, in pyarrow._dataset.Dataset.to_table
File "pyarrow/_dataset.pyx", line 2818, in pyarrow._dataset.Scanner.to_table
File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 115, in pyarrow.lib.check_status
OSError: AWS Error NETWORK_CONNECTION during GetObject operation: curlCode: 43, A libcurl function was given a bad argument
```
We were trying to find the reason why it happens, but it is very random. Can you help to understand where actually the problem with libcurl can be?
### Component(s)
Python
Contributor guide
Research direction
Start with the _load_partition entry point in storage.py at line 794 and the pyarrow dataset call shown in the traceback. Reproduce the intermittent S3 GetObject failure if possible and trace the curlCode 43 path; done means identifying a reproducible cause and a confirmed fix or clear upstream diagnosis.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100