apache / apache/iceberg-python

[Bug] PyArrowFileIO fails to propagate s3.ssl.ca-cert to pyarrow.fs.S3FileSystem tls_ca_file_path

Open Beginner friendly
#3,866 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.1k
Forks
581
Avg merge
1d 17h
Merged PRs (30d)
78

Description

### Apache Iceberg version

0.11.0 (latest release)

### Please describe the bug 🐞

When configuring PyIceberg with a REST catalog (such as Apache Polaris) connected to an S3-compatible store using internal or self-signed certificates, providing "s3.ssl.ca-cert" in the catalog configuration does not take effect for PyArrow file operations.

PyIceberg's PyArrowFileIO instantiates pyarrow.fs.S3FileSystem without forwarding the custom CA certificate path to PyArrow's tls_ca_file_path argument. As a result, operations that write or read Parquet files fail with curlCode: 60 (SSL verification failure in PyArrow's underlying AWS C++ SDK / libcurl engine).

### Willingness to contribute

- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at PyIceberg's PyArrowFileIO entry point where pyarrow.fs.S3FileSystem is instantiated, then trace how the catalog's "s3.ssl.ca-cert" setting is read. Verify the relevant PyArrow argument is covered for both Parquet reads and writes, and reproduce the issue with an internal or self-signed certificate to confirm the SSL verification failure is resolved.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, data-engineering
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.