apache / apache/arrow

[Python] Rewrite pyarrow.jvm using the C data interface

Open
#44,860 16 comments 0 reactions 0 assignees View on GitHub
Component: Python
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

The `pyarrow.jvm` is currently a custom-written bridge between PyArrow and Arrow Java, with limited datatype support. Now that Java implements the C data interface (see ARROW-12965), we should be able to simplify the code while making it more general.

Also, we should reenable the `conda-python-jpype` build somewhere, for example in the Crossbow nightly builds.

**Reporter**: [Antoine Pitrou](https://issues.apache.org/jira/browse/ARROW-14319) / @pitrou
#### Related issues:
- [[Java] Java implementation of Arrow C data interface](https://github.com/apache/arrow/issues/28685) (is related to)

**Note**: *This issue was originally created as [ARROW-14319](https://issues.apache.org/jira/browse/ARROW-14319). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by examining the current pyarrow.jvm bridge and the related Java C data interface issue ARROW-12965. Compare the bridge with Java's interface implementation, then inspect how the conda-python-jpype build is configured for possible Crossbow nightly reactivation. Done means the bridge uses the C data interface with broader datatype support and the build is enabled again.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, python
Domain
data-engineering
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.