apache / apache/iceberg-python
[feature] Investigate integrations leveraging the PyCapsule protocol
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 581
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 78
Description
### Feature Request / Improvement
Context: https://github.com/apache/iceberg-python/pull/1614#issuecomment-2641089912
Copying the comment over:
Separately, rather than adding more library-specific conversion code, it might make sense for pyiceberg to start leveraging the [PyCapsule protocol](https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html) to allow any third party library (dataframe or otherwise) that supports Arrow data to seamlessly consume pyiceberg constructs.
Polars already supports the PyCapsule interface. See https://docs.pola.rs/user-guide/misc/arrow/#using-the-arrow-pycapsule-interface for details.
Implementing the interface on e.g. pyiceberg tables would allow them to be passed directly to dataframe init in polars, just like you can do a pyarrow table today. It also doesn't assume anything about polars support/doesn't add a dependency on polars.
cc @corleyma if you would like to provide more context :)
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.