[Python] AWS Error NETWORK_CONNECTION during HeadObject operation: curlCode: 6, Couldn't resolve host name
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
Hi there!
We are using the PyArrow library to read files from an S3 bucket, and we're encountering an intermittent error:
`OSError: When reading information for key '' in bucket '': AWS Error NETWORK_CONNECTION during HeadObject operation: curlCode: 6, Couldn't resolve host name`
Please note that this error doesn't occur consistently, and the S3 bucket path is valid.
The reference code we're using is as follows:
```
import pyarrow as pa
import pyarrow.json as pj
uri = "s3://my-bucket/my-prefix/foo.json"
fs, path = pa.fs.FileSystem.from_uri(uri)
with fs.open_input_file(path) as f:
tbl = pj.read_json(f)
```
Error Details:
```
2024-03-12T13:06:05.616-07:00 [7]: with fs.open_input_file(path) as f:
2024-03-12T13:06:05.616-07:00 [7]: File "pyarrow/_fs.pyx", line 780, in pyarrow._fs.FileSystem.open_input_file
2024-03-12T13:06:05.616-07:00 [7]: File "pyarrow/error.pxi", line 154, in pyarrow.lib.pyarrow_internal_check_status
2024-03-12T13:06:05.616-07:00 [7]: File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status
2024-03-12T13:06:05.616-07:00 [7]:OSError: When reading information for key '' in bucket '': AWS Error NETWORK_CONNECTION during HeadObject operation: curlCode: 6, Couldn't resolve host name
```
Could you please provide any suggestions on how to handle such intermittent network connectivity errors while reading from S3?
### Component(s)
Parquet, Python
Contributor guide
Assessment
This issue has not been assessed yet.