apache / apache/iceberg-python
[Bug] PyArrowFileIO fails to propagate s3.ssl.ca-cert to pyarrow.fs.S3FileSystem tls_ca_file_path
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 581
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 78
Description
### Apache Iceberg version
0.11.0 (latest release)
### Please describe the bug 🐞
When configuring PyIceberg with a REST catalog (such as Apache Polaris) connected to an S3-compatible store using internal or self-signed certificates, providing "s3.ssl.ca-cert" in the catalog configuration does not take effect for PyArrow file operations.
PyIceberg's PyArrowFileIO instantiates pyarrow.fs.S3FileSystem without forwarding the custom CA certificate path to PyArrow's tls_ca_file_path argument. As a result, operations that write or read Parquet files fail with curlCode: 60 (SSL verification failure in PyArrow's underlying AWS C++ SDK / libcurl engine).
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at PyIceberg's PyArrowFileIO entry point where pyarrow.fs.S3FileSystem is instantiated, then trace how the catalog's "s3.ssl.ca-cert" setting is read. Verify the relevant PyArrow argument is covered for both Parquet reads and writes, and reproduce the issue with an internal or self-signed certificate to confirm the SSL verification failure is resolved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, data-engineering
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100