apache / apache/iceberg-python
[Bug] PyArrowFileIO fails to propagate s3.ssl.ca-cert to pyarrow.fs.S3FileSystem tls_ca_file_path
- Lenguaje dominante
- Python
- Estrellas
- 1.1k
- Forks
- 581
- Merge medio
- 1 d 17 h
- PR fusionados (30 d)
- 77
Descripción
### Apache Iceberg version
0.11.0 (latest release)
### Please describe the bug 🐞
When configuring PyIceberg with a REST catalog (such as Apache Polaris) connected to an S3-compatible store using internal or self-signed certificates, providing "s3.ssl.ca-cert" in the catalog configuration does not take effect for PyArrow file operations.
PyIceberg's PyArrowFileIO instantiates pyarrow.fs.S3FileSystem without forwarding the custom CA certificate path to PyArrow's tls_ca_file_path argument. As a result, operations that write or read Parquet files fail with curlCode: 60 (SSL verification failure in PyArrow's underlying AWS C++ SDK / libcurl engine).
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
Guía de contribución
No hay ninguna guía de contribución indexada para este repositorio
Línea de trabajo
Start at PyIceberg's PyArrowFileIO entry point where pyarrow.fs.S3FileSystem is instantiated, then trace how the catalog's "s3.ssl.ca-cert" setting is read. Verify the relevant PyArrow argument is covered for both Parquet reads and writes, and reproduce the issue with an internal or self-signed certificate to confirm the SSL verification failure is resolved.
Escrito por el modelo de indexación a partir del texto del issue.
Evaluación
- Stack tecnológico
- aws, python
- Área
- cloud, data-engineering
- Tipo de issue
- Error
- Dificultad
- 2/5
- Tiempo estimado
- 1-3 horas
- Estado de actividad
- Activo
- Claridad
- Bien especificado
- Aptitud para principiantes
- 76/100