apache / apache/arrow-java

[Java/Python] in-process vector sharing from Java to Python

Abierto
#420 3 comentarios 0 reacciones 0 asignados Ver en GitHub
Type: enhancement
Lenguaje dominante
Java
Estrellas
94
Forks
152
Merge medio
3 d 16 h
PR fusionados (30 d)
11

Descripción

Currently we seem to use in all applications of Arrow the IPC capabilities to move data between a Java process and a Python process. While this is 0-serialization, it is not zero-copy. By taking the address and offset, we can already create Python buffers from Java buffers: https://github.com/apache/arrow/pull/1693. This is still a very low-level interface and we should provide the user with:

- A guide on how to load Apache Arrow java libraries in Python (either through a fat-jar that was shipped with Arrow or how he should integrate it into its Java packaging)
- `pyarrow.Array.from_jvm`, `pyarrow.RecordBatch.from_jvm`, … functions that take the respective Java objects and emit Python objects. These Python objects should also ensure that the underlying memory regions are kept alive as long as the Python objects exist.

This issue can also be used as a tracker for the various sub-tasks that will need to be done to complete this rather large milestone.

**Reporter**: [Uwe Korn](https://issues.apache.org/jira/browse/ARROW-2249) / @xhochy
#### Related issues:
- [[Python] Create buffer from address, size and base](https://github.com/apache/arrow/issues/18211) (is blocked by)
- [[Java/Python] Add unit test for pyarrow.timeX types in Array.from_jvm](https://github.com/apache/arrow/issues/44875) (is a parent of)
- [[Java/Python]  Add unit test for pyarrow.decimal128 in Array.from_jvm](https://github.com/apache/arrow/issues/44874) (is a parent of)
- [[Java/Python] Support VarCharVector / StringArray in pyarrow.Array.from_jvm](https://github.com/apache/arrow/issues/44873) (is a parent of)
- [[Java/Python] Complex type conversion in pyarrow.Field.from_jvm](https://github.com/apache/arrow/issues/19009) (is a parent of)
- [[Java/Python] Add support for dictionary type to pyarrow.Field.from_jvm](https://github.com/apache/arrow/issues/15719) (is a parent of)
- [[Java/Python] Add pyarrow.{Array,Field}.from_jvm / jvm_buffer](https://github.com/apache/arrow/issues/19008) (is a parent of)
- [[Java] Add method overload for VarCharVector.set(int,String)](https://github.com/apache/arrow/issues/19004) (is related to)
- [[Java] Java implementation of Arrow C data interface](https://github.com/apache/arrow/issues/28685) (is related to)

**Note**: *This issue was originally created as [ARROW-2249](https://issues.apache.org/jira/browse/ARROW-2249). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Guía de contribución

Abrir la guía de contribución

Línea de trabajo

Comience con los puntos de entrada propuestos pyarrow.Array.from_jvm y pyarrow.RecordBatch.from_jvm y, después, revise el issue relacionado con la compatibilidad con pyarrow.Array, Field y jvm_buffer. Consulte la documentación de migración y el issue del búfer bloqueado antes de elegir una subtarea. Para darlo por terminado, debe existir una ruta de conversión definida de Java a Python con gestión zero-copy de la vida útil de la memoria.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
java, python
Área
data, distributed-systems
Tipo de issue
Nueva funcionalidad
Dificultad
5/5
Tiempo estimado
Más de una semana
Estado de actividad
Estancado
Claridad
Necesita aclaración
Aptitud para principiantes
25/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.