[Python][Packaging] S3FileSystem curl error when using `localstack`-created S3 bucket or custom ca-certificate
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
I'm trying to use a [localstack](https://docs.localstack.cloud/getting-started/quickstart/)-created S3 bucket as way to test my application without interacting with S3.
To do that I launch an S3 endpoint using `localstack start -d` and create my PyArrow S3FS using:
```
s3_fs = fs.S3FileSystem(endpoint_override="localhost:4566")
```
When I try interacting with files on the simulated bucket however I get the following:
```
In [223]: nrows = pq.read_metadata(f"{file_bucket}/{file_key}", filesystem=s3_fs).num_rows
---------------------------------------------------------------------------
OSError Traceback (most recent call last)
in
----> 1 nrows = pq.read_metadata(f"{file_bucket}/{file_key}", filesystem=s3_fs).num_rows
/[...]/lib/python3.7/site-packages/pyarrow/parquet/core.py in read_metadata(where, memory_map, decryption_properties, filesystem)
3479 file_ctx = nullcontext()
3480 if filesystem is not None:
-> 3481 file_ctx = where = filesystem.open_input_file(where)
3482
3483 with file_ctx:
[...]/python3.7/site-packages/pyarrow/_fs.pyx in pyarrow._fs.FileSystem.open_input_file()
[...]/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()
[...]/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()
OSError: When reading information for key 'redacted/path/to/file' in bucket 'example-bucket': AWS Error NETWO
RK_CONNECTION during HeadObject operation: curlCode: 60, SSL peer certificate or SSH remote key was not OK
```
Another user seems to have the same problem when using on-prem S3, and had to use `s3fs` along with `PyFileSystem, FSSpecHandler` to resolve it: https://discuss.ray.io/t/ssl-peer-certificate-or-ssh-remote-key-was-not-ok/11091/2
Fully reproducible example:
```
pip install localstack awscli-local pyarrow
localstack start -d
awslocal s3 mb example-bucket
python <", line 4, in
File "[.../]lib/python3.9/site-packages/pyarrow/parquet/core.py", line 3481, in read_metadata
file_ctx = where = filesystem.open_input_file(where)
File "pyarrow/_fs.pyx", line 770, in pyarrow._fs.FileSystem.open_input_file
File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 115, in pyarrow.lib.check_status
OSError: When reading information for key 'example.parquet' in bucket 'example-bucket': AWS Error NETWORK_CONNECTION during HeadObject operation: curlCode: 60, SSL peer certificate or SSH remote key was not OK
```
Possibly related: https://issues.apache.org/jira/browse/ARROW-9261
Tested on Pyarrow 10.0.1, 12.0.1.
### Component(s)
Python
Contributor guide
Assessment
This issue has not been assessed yet.