[Python] Rewrite pyarrow.jvm using the C data interface
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
The `pyarrow.jvm` is currently a custom-written bridge between PyArrow and Arrow Java, with limited datatype support. Now that Java implements the C data interface (see ARROW-12965), we should be able to simplify the code while making it more general.
Also, we should reenable the `conda-python-jpype` build somewhere, for example in the Crossbow nightly builds.
**Reporter**: [Antoine Pitrou](https://issues.apache.org/jira/browse/ARROW-14319) / @pitrou
#### Related issues:
- [[Java] Java implementation of Arrow C data interface](https://github.com/apache/arrow/issues/28685) (is related to)
**Note**: *This issue was originally created as [ARROW-14319](https://issues.apache.org/jira/browse/ARROW-14319). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start by examining the current pyarrow.jvm bridge and the related Java C data interface issue ARROW-12965. Compare the bridge with Java's interface implementation, then inspect how the conda-python-jpype build is configured for possible Crossbow nightly reactivation. Done means the bridge uses the C data interface with broader datatype support and the build is enabled again.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, python
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100