apache / apache/iceberg-python

Allow Arrow Capsule Interface

Abierto
#2,680 5 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Python
Estrellas
1.1k
Forks
581
Merge medio
1 d 17 h
PR fusionados (30 d)
78

Descripción

### Apache Iceberg version

0.10.0 (latest release)

### Please describe the bug 🐞

Due to how `iceberg-python` does checks in certain places, I can't use libraries such as [`arro3`](https://github.com/kylebarron/arro3) or `polars` without having to do a conversion and include `pyarrow` as a dependency. Here is such a case in `table/__init__.py`:

```python
def append(self, df: pa.Table, snapshot_properties: Dict[str, str] = EMPTY_DICT, branch: Optional[str] = MAIN_BRANCH) -> None:
"""
Shorthand API for appending a PyArrow table to a table transaction.

Args:
df: The Arrow dataframe that will be appended to overwrite the table
snapshot_properties: Custom properties to be added to the snapshot summary
branch: Branch Reference to run the append operation
"""
try:
import pyarrow as pa
except ModuleNotFoundError as e:
raise ModuleNotFoundError("For writes PyArrow needs to be installed") from e

from pyiceberg.io.pyarrow import _check_pyarrow_schema_compatible, _dataframe_to_data_files

if not isinstance(df, pa.Table):
raise ValueError(f"Expected PyArrow table, got: {df}")
```

Can this be updated to use the capsule interface: https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html ?

I can create a patch if this is something that will be accepted. Sorry for the new account, due to employer issues I can't use my "regular" one.

### Willingness to contribute

- [x] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Línea de trabajo

Comienza leyendo la implementación de append en table/__init__.py y los helpers de compatibilidad y de archivos de datos de pyiceberg.io.pyarrow a los que hace referencia; después, compara sus comprobaciones de entrada con Arrow PyCapsule Interface. Se considera terminado cuando productores de Arrow compatibles, como arro3 o polars, pueden utilizarse sin conversión ni una dependencia obligatoria de pyarrow.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
python
Área
data
Tipo de issue
Error
Dificultad
3/5
Tiempo estimado
1-2 días
Estado de actividad
Tranquilo
Claridad
Bastante claro
Aptitud para principiantes
64/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.