[Java/Python] in-process vector sharing from Java to Python
- 主要语言
- Java
- 星标
- 94
- 派生
- 152
- 平均合并
- 3 天 16 小时
- 30 天内合并 PR
- 11
描述
Currently we seem to use in all applications of Arrow the IPC capabilities to move data between a Java process and a Python process. While this is 0-serialization, it is not zero-copy. By taking the address and offset, we can already create Python buffers from Java buffers: https://github.com/apache/arrow/pull/1693. This is still a very low-level interface and we should provide the user with:
- A guide on how to load Apache Arrow java libraries in Python (either through a fat-jar that was shipped with Arrow or how he should integrate it into its Java packaging)
- `pyarrow.Array.from_jvm`, `pyarrow.RecordBatch.from_jvm`, … functions that take the respective Java objects and emit Python objects. These Python objects should also ensure that the underlying memory regions are kept alive as long as the Python objects exist.
This issue can also be used as a tracker for the various sub-tasks that will need to be done to complete this rather large milestone.
**Reporter**: [Uwe Korn](https://issues.apache.org/jira/browse/ARROW-2249) / @xhochy
#### Related issues:
- [[Python] Create buffer from address, size and base](https://github.com/apache/arrow/issues/18211) (is blocked by)
- [[Java/Python] Add unit test for pyarrow.timeX types in Array.from_jvm](https://github.com/apache/arrow/issues/44875) (is a parent of)
- [[Java/Python] Add unit test for pyarrow.decimal128 in Array.from_jvm](https://github.com/apache/arrow/issues/44874) (is a parent of)
- [[Java/Python] Support VarCharVector / StringArray in pyarrow.Array.from_jvm](https://github.com/apache/arrow/issues/44873) (is a parent of)
- [[Java/Python] Complex type conversion in pyarrow.Field.from_jvm](https://github.com/apache/arrow/issues/19009) (is a parent of)
- [[Java/Python] Add support for dictionary type to pyarrow.Field.from_jvm](https://github.com/apache/arrow/issues/15719) (is a parent of)
- [[Java/Python] Add pyarrow.{Array,Field}.from_jvm / jvm_buffer](https://github.com/apache/arrow/issues/19008) (is a parent of)
- [[Java] Add method overload for VarCharVector.set(int,String)](https://github.com/apache/arrow/issues/19004) (is related to)
- [[Java] Java implementation of Arrow C data interface](https://github.com/apache/arrow/issues/28685) (is related to)
**Note**: *This issue was originally created as [ARROW-2249](https://issues.apache.org/jira/browse/ARROW-2249). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
贡献指南
调研方向
从提议的入口点 pyarrow.Array.from_jvm 和 pyarrow.RecordBatch.from_jvm 开始,然后查看与支持 pyarrow.Array、Field 和 jvm_buffer 相关的 issue。在选择子任务之前,检查迁移文档和被阻塞的 buffer issue。完成的要求是定义 Java 到 Python 的转换路径,并处理 zero-copy 内存生命周期。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- java, python
- 领域
- data, distributed-systems
- Issue 类型
- 功能
- 难度
- 5/5
- 预计耗时
- 一周以上
- 活跃度
- 停滞
- 描述清晰度
- 需要澄清
- 新手友好度
- 25/100