[Java] Implement "ArrowBufReadChannel" abstraction and alternate MessageSerializer that uses this
- Langage dominant
- Java
- Étoiles
- 94
- Forks
- 152
- Merge moyen
- 3 j 16 h
- PR mergées (30 j)
- 11
Description
The current MessageSerializer implementation is wasteful when used to read an IPC payload that is already in-memory in an `ArrowBuf`. In particular, reads out of a `ReadChannel` require memory allocation
- https://github.com/apache/arrow/blob/master/java/vector/src/main/java/org/apache/arrow/vector/ipc/message/MessageSerializer.java#L569
- https://github.com/apache/arrow/blob/master/java/vector/src/main/java/org/apache/arrow/vector/ipc/message/MessageSerializer.java#L290
In C++, we have abstracted memory allocation out of the IPC read path so that zero-copy is possible. I suggest that a similar mechanism can be developed for Java to improve deserialization performance for in-memory messages. The new interface would return `ArrowBuf` when performing reads, which could be zero-copy when possible, but when not the current strategy of allocate-copy could be used
**Reporter**: [Wes McKinney](https://issues.apache.org/jira/browse/ARROW-3192) / @wesm
**Note**: *This issue was originally created as [ARROW-3192](https://issues.apache.org/jira/browse/ARROW-3192). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Guide de contribution
Ouvrir le guide de contribution
Piste de recherche
Lisez java/vector/src/main/java/org/apache/arrow/vector/ipc/message/MessageSerializer.java aux lignes référencées 569 et 290 pour comprendre le chemin actuel d’allocation de ReadChannel. Étudiez l’abstraction ArrowBufReadChannel proposée et le MessageSerializer alternatif, avec des lectures zero-copy pour les payloads ArrowBuf en mémoire et un fallback d’allocation-copie comme critères d’achèvement.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- java
- Domaine
- data-engineering
- Type d'issue
- Fonctionnalité
- Difficulté
- 5/5
- Temps estimé
- Plus d'une semaine
- Activité
- À l'abandon
- Clarté
- Plutôt claire
- Accessibilité débutants
- 35/100