[Java] Implement "ArrowBufReadChannel" abstraction and alternate MessageSerializer that uses this
- Linguagem predominante
- Java
- Estrelas
- 94
- Forks
- 152
- Merge médio
- 3d 16h
- PRs com merge (30d)
- 11
Descrição
The current MessageSerializer implementation is wasteful when used to read an IPC payload that is already in-memory in an `ArrowBuf`. In particular, reads out of a `ReadChannel` require memory allocation
- https://github.com/apache/arrow/blob/master/java/vector/src/main/java/org/apache/arrow/vector/ipc/message/MessageSerializer.java#L569
- https://github.com/apache/arrow/blob/master/java/vector/src/main/java/org/apache/arrow/vector/ipc/message/MessageSerializer.java#L290
In C++, we have abstracted memory allocation out of the IPC read path so that zero-copy is possible. I suggest that a similar mechanism can be developed for Java to improve deserialization performance for in-memory messages. The new interface would return `ArrowBuf` when performing reads, which could be zero-copy when possible, but when not the current strategy of allocate-copy could be used
**Reporter**: [Wes McKinney](https://issues.apache.org/jira/browse/ARROW-3192) / @wesm
**Note**: *This issue was originally created as [ARROW-3192](https://issues.apache.org/jira/browse/ARROW-3192). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Guia de contribuição
Direção de pesquisa
Leia java/vector/src/main/java/org/apache/arrow/vector/ipc/message/MessageSerializer.java nas linhas referenciadas 569 e 290 para entender o caminho atual de alocação de ReadChannel. Investigue a abstração ArrowBufReadChannel proposta e o MessageSerializer alternativo, tendo leituras zero-copy para payloads ArrowBuf em memória e um fallback de alocação e cópia como critérios de conclusão.
Escrita pelo modelo de indexação a partir do texto da issue.
Avaliação
- Stack de tecnologia
- java
- Domínio
- data-engineering
- Tipo de issue
- Funcionalidade
- Dificuldade
- 5/5
- Tempo estimado
- Mais de uma semana
- Status de atividade
- Estagnada
- Clareza
- Razoavelmente clara
- Facilidade para iniciantes
- 35/100