apache / apache/datafusion-python

Remove pyarrow as required dependency, relying on Arrow PyCapsule Interface

オープン
#1,227 コメント 14 件 リアクション 2 件 担当者 0 名 GitHub で見る
enhancement
主要言語
Python
スター
604
フォーク
174
平均マージ
1日 7時間
マージ済み PR(30日)
4

説明

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

PyArrow is a massive dependency. Unpacked, it tends to be >100MB in size, and, until the latest versions (I think?) also required numpy as its own non-optional dependency.

It's also, in effect the only current dependency
https://github.com/apache/datafusion-python/blob/f0bbad7543717c5f08ba2acb92d42c9d30fd2355/pyproject.toml#L46

It would be great if we could remove it, and that would greatly lessen the minimal environment size for datafusion python.

[Many other Python Arrow libraries](https://github.com/apache/arrow/issues/39195#issuecomment-2245718008) implement the PyCapsule Interface, so the user can use nanoarrow, arro3, Polars, DuckDB, etc, or pyarrow. Whatever is best for them.

**Describe the solution you'd like**

The Arrow PyCapsule Interface is a lightweight, decentralized protocol for sharing Arrow data between Python libraries. We already implement the PyCapsule Interface, so it's just a matter of removing places where we hard-code use of pyarrow.

**Describe alternatives you've considered**

Keep pyarrow dependency.

**Additional context**

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。