apache / apache/iceberg-python
Allow Arrow Capsule Interface
- Lenguaje dominante
- Python
- Estrellas
- 1.1k
- Forks
- 581
- Merge medio
- 1 d 17 h
- PR fusionados (30 d)
- 78
Descripción
### Apache Iceberg version
0.10.0 (latest release)
### Please describe the bug 🐞
Due to how `iceberg-python` does checks in certain places, I can't use libraries such as [`arro3`](https://github.com/kylebarron/arro3) or `polars` without having to do a conversion and include `pyarrow` as a dependency. Here is such a case in `table/__init__.py`:
```python
def append(self, df: pa.Table, snapshot_properties: Dict[str, str] = EMPTY_DICT, branch: Optional[str] = MAIN_BRANCH) -> None:
"""
Shorthand API for appending a PyArrow table to a table transaction.
Args:
df: The Arrow dataframe that will be appended to overwrite the table
snapshot_properties: Custom properties to be added to the snapshot summary
branch: Branch Reference to run the append operation
"""
try:
import pyarrow as pa
except ModuleNotFoundError as e:
raise ModuleNotFoundError("For writes PyArrow needs to be installed") from e
from pyiceberg.io.pyarrow import _check_pyarrow_schema_compatible, _dataframe_to_data_files
if not isinstance(df, pa.Table):
raise ValueError(f"Expected PyArrow table, got: {df}")
```
Can this be updated to use the capsule interface: https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html ?
I can create a patch if this is something that will be accepted. Sorry for the new account, due to employer issues I can't use my "regular" one.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
Guía de contribución
No hay ninguna guía de contribución indexada para este repositorio
Línea de trabajo
Comienza leyendo la implementación de append en table/__init__.py y los helpers de compatibilidad y de archivos de datos de pyiceberg.io.pyarrow a los que hace referencia; después, compara sus comprobaciones de entrada con Arrow PyCapsule Interface. Se considera terminado cuando productores de Arrow compatibles, como arro3 o polars, pueden utilizarse sin conversión ni una dependencia obligatoria de pyarrow.
Escrito por el modelo de indexación a partir del texto del issue.
Evaluación
- Stack tecnológico
- python
- Área
- data
- Tipo de issue
- Error
- Dificultad
- 3/5
- Tiempo estimado
- 1-2 días
- Estado de actividad
- Tranquilo
- Claridad
- Bastante claro
- Aptitud para principiantes
- 64/100