apache / apache/arrow

[Python] AWS Error NETWORK_CONNECTION during HeadObject operation: curlCode: 6, Couldn't resolve host name

Open
#40,539 7 comments 7 reactions 0 assignees View on GitHub
Component: Parquet Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Hi there!

We are using the PyArrow library to read files from an S3 bucket, and we're encountering an intermittent error:

`OSError: When reading information for key '' in bucket '': AWS Error NETWORK_CONNECTION during HeadObject operation: curlCode: 6, Couldn't resolve host name`

Please note that this error doesn't occur consistently, and the S3 bucket path is valid.

The reference code we're using is as follows:
```
import pyarrow as pa
import pyarrow.json as pj

uri = "s3://my-bucket/my-prefix/foo.json"

fs, path = pa.fs.FileSystem.from_uri(uri)

with fs.open_input_file(path) as f:
tbl = pj.read_json(f)
```

Error Details:
```
2024-03-12T13:06:05.616-07:00 [7]: with fs.open_input_file(path) as f:

2024-03-12T13:06:05.616-07:00 [7]: File "pyarrow/_fs.pyx", line 780, in pyarrow._fs.FileSystem.open_input_file

2024-03-12T13:06:05.616-07:00 [7]: File "pyarrow/error.pxi", line 154, in pyarrow.lib.pyarrow_internal_check_status

2024-03-12T13:06:05.616-07:00 [7]: File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status

2024-03-12T13:06:05.616-07:00 [7]:OSError: When reading information for key '' in bucket '': AWS Error NETWORK_CONNECTION during HeadObject operation: curlCode: 6, Couldn't resolve host name

```

Could you please provide any suggestions on how to handle such intermittent network connectivity errors while reading from S3?

### Component(s)

Parquet, Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.