apache / apache/iceberg-python
Allow Arrow Capsule Interface
- Langage dominant
- Python
- Étoiles
- 1.1k
- Forks
- 581
- Merge moyen
- 1 j 17 h
- PR mergées (30 j)
- 78
Description
### Apache Iceberg version
0.10.0 (latest release)
### Please describe the bug 🐞
Due to how `iceberg-python` does checks in certain places, I can't use libraries such as [`arro3`](https://github.com/kylebarron/arro3) or `polars` without having to do a conversion and include `pyarrow` as a dependency. Here is such a case in `table/__init__.py`:
```python
def append(self, df: pa.Table, snapshot_properties: Dict[str, str] = EMPTY_DICT, branch: Optional[str] = MAIN_BRANCH) -> None:
"""
Shorthand API for appending a PyArrow table to a table transaction.
Args:
df: The Arrow dataframe that will be appended to overwrite the table
snapshot_properties: Custom properties to be added to the snapshot summary
branch: Branch Reference to run the append operation
"""
try:
import pyarrow as pa
except ModuleNotFoundError as e:
raise ModuleNotFoundError("For writes PyArrow needs to be installed") from e
from pyiceberg.io.pyarrow import _check_pyarrow_schema_compatible, _dataframe_to_data_files
if not isinstance(df, pa.Table):
raise ValueError(f"Expected PyArrow table, got: {df}")
```
Can this be updated to use the capsule interface: https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html ?
I can create a patch if this is something that will be accepted. Sorry for the new account, due to employer issues I can't use my "regular" one.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
Guide de contribution
Aucun guide de contribution indexé pour ce dépôt
Piste de recherche
Commencez par lire l’implémentation de append dans table/__init__.py ainsi que les utilitaires de compatibilité et de fichiers de données de pyiceberg.io.pyarrow référencés, puis comparez leurs vérifications des entrées avec Arrow PyCapsule Interface. Le travail est terminé lorsque des producteurs Arrow pris en charge tels que arro3 ou polars peuvent être utilisés sans conversion ni dépendance obligatoire à pyarrow.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- python
- Domaine
- data
- Type d'issue
- Bug
- Difficulté
- 3/5
- Temps estimé
- 1-2 jours
- Activité
- Calme
- Clarté
- Plutôt claire
- Accessibilité débutants
- 64/100