[Java] Implement "ArrowBufReadChannel" abstraction and alternate MessageSerializer that uses this
- Lenguaje dominante
- Java
- Estrellas
- 94
- Forks
- 152
- Merge medio
- 3 d 16 h
- PR fusionados (30 d)
- 11
Descripción
The current MessageSerializer implementation is wasteful when used to read an IPC payload that is already in-memory in an `ArrowBuf`. In particular, reads out of a `ReadChannel` require memory allocation
- https://github.com/apache/arrow/blob/master/java/vector/src/main/java/org/apache/arrow/vector/ipc/message/MessageSerializer.java#L569
- https://github.com/apache/arrow/blob/master/java/vector/src/main/java/org/apache/arrow/vector/ipc/message/MessageSerializer.java#L290
In C++, we have abstracted memory allocation out of the IPC read path so that zero-copy is possible. I suggest that a similar mechanism can be developed for Java to improve deserialization performance for in-memory messages. The new interface would return `ArrowBuf` when performing reads, which could be zero-copy when possible, but when not the current strategy of allocate-copy could be used
**Reporter**: [Wes McKinney](https://issues.apache.org/jira/browse/ARROW-3192) / @wesm
**Note**: *This issue was originally created as [ARROW-3192](https://issues.apache.org/jira/browse/ARROW-3192). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Guía de contribución
Línea de trabajo
Lea java/vector/src/main/java/org/apache/arrow/vector/ipc/message/MessageSerializer.java en las líneas referenciadas 569 y 290 para comprender la ruta actual de asignación de ReadChannel. Investigue la abstracción ArrowBufReadChannel propuesta y el MessageSerializer alternativo, con lecturas zero-copy para payloads de ArrowBuf en memoria y un fallback de asignación y copia como criterios de finalización.
Escrito por el modelo de indexación a partir del texto del issue.
Evaluación
- Stack tecnológico
- java
- Área
- data-engineering
- Tipo de issue
- Nueva funcionalidad
- Dificultad
- 5/5
- Tiempo estimado
- Más de una semana
- Estado de actividad
- Estancado
- Claridad
- Bastante claro
- Aptitud para principiantes
- 35/100