apache / apache/iceberg-python
[Bug] PyArrowFileIO fails to propagate s3.ssl.ca-cert to pyarrow.fs.S3FileSystem tls_ca_file_path
- Linguagem predominante
- Python
- Estrelas
- 1.1k
- Forks
- 581
- Merge médio
- 1d 13h
- PRs com merge (30d)
- 76
Descrição
### Apache Iceberg version
0.11.0 (latest release)
### Please describe the bug 🐞
When configuring PyIceberg with a REST catalog (such as Apache Polaris) connected to an S3-compatible store using internal or self-signed certificates, providing "s3.ssl.ca-cert" in the catalog configuration does not take effect for PyArrow file operations.
PyIceberg's PyArrowFileIO instantiates pyarrow.fs.S3FileSystem without forwarding the custom CA certificate path to PyArrow's tls_ca_file_path argument. As a result, operations that write or read Parquet files fail with curlCode: 60 (SSL verification failure in PyArrow's underlying AWS C++ SDK / libcurl engine).
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
Guia de contribuição
Nenhum guia de contribuição indexado para este repositório
Direção de pesquisa
Start at PyIceberg's PyArrowFileIO entry point where pyarrow.fs.S3FileSystem is instantiated, then trace how the catalog's "s3.ssl.ca-cert" setting is read. Verify the relevant PyArrow argument is covered for both Parquet reads and writes, and reproduce the issue with an internal or self-signed certificate to confirm the SSL verification failure is resolved.
Escrita pelo modelo de indexação a partir do texto da issue.
Avaliação
- Stack de tecnologia
- aws, python
- Domínio
- cloud, data-engineering
- Tipo de issue
- Bug
- Dificuldade
- 2/5
- Tempo estimado
- 1-3 horas
- Status de atividade
- Ativa
- Clareza
- Claramente especificada
- Facilidade para iniciantes
- 76/100