apache / apache/datafusion-python
Remove pyarrow as required dependency, relying on Arrow PyCapsule Interface
- Ngôn ngữ chính
- Python
- Star
- 604
- Fork
- 174
- Merge trung bình
- 1 ngày 7 giờ
- Pull request đã merge (30 ngày)
- 4
Mô tả
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
PyArrow is a massive dependency. Unpacked, it tends to be >100MB in size, and, until the latest versions (I think?) also required numpy as its own non-optional dependency.
It's also, in effect the only current dependency
https://github.com/apache/datafusion-python/blob/f0bbad7543717c5f08ba2acb92d42c9d30fd2355/pyproject.toml#L46
It would be great if we could remove it, and that would greatly lessen the minimal environment size for datafusion python.
[Many other Python Arrow libraries](https://github.com/apache/arrow/issues/39195#issuecomment-2245718008) implement the PyCapsule Interface, so the user can use nanoarrow, arro3, Polars, DuckDB, etc, or pyarrow. Whatever is best for them.
**Describe the solution you'd like**
The Arrow PyCapsule Interface is a lightweight, decentralized protocol for sharing Arrow data between Python libraries. We already implement the PyCapsule Interface, so it's just a matter of removing places where we hard-code use of pyarrow.
**Describe alternatives you've considered**
Keep pyarrow dependency.
**Additional context**
Hướng dẫn đóng góp
Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này
Hướng nghiên cứu
Start by inspecting pyproject.toml at line 46 and locating the remaining hard-coded pyarrow uses in the Python bindings. Check how the existing Arrow PyCapsule Interface support is used across those paths. Done means pyarrow is no longer required while the affected paths continue to support Arrow data providers.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Đánh giá
- Công nghệ
- python
- Lĩnh vực
- data-engineering
- Loại issue
- Tính năng
- Độ khó
- 4/5
- Thời gian dự kiến
- 3-5 ngày
- Mức độ hoạt động
- Đình trệ
- Độ rõ ràng
- Khá rõ ràng
- Mức phù hợp với người mới
- 35/100