[Java] Implement "ArrowBufReadChannel" abstraction and alternate MessageSerializer that uses this
- Dominant language
- Java
- Stars
- 94
- Forks
- 152
- Avg merge
- 3d 16h
- Merged PRs (30d)
- 11
Description
The current MessageSerializer implementation is wasteful when used to read an IPC payload that is already in-memory in an `ArrowBuf`. In particular, reads out of a `ReadChannel` require memory allocation
- https://github.com/apache/arrow/blob/master/java/vector/src/main/java/org/apache/arrow/vector/ipc/message/MessageSerializer.java#L569
- https://github.com/apache/arrow/blob/master/java/vector/src/main/java/org/apache/arrow/vector/ipc/message/MessageSerializer.java#L290
In C++, we have abstracted memory allocation out of the IPC read path so that zero-copy is possible. I suggest that a similar mechanism can be developed for Java to improve deserialization performance for in-memory messages. The new interface would return `ArrowBuf` when performing reads, which could be zero-copy when possible, but when not the current strategy of allocate-copy could be used
**Reporter**: [Wes McKinney](https://issues.apache.org/jira/browse/ARROW-3192) / @wesm
**Note**: *This issue was originally created as [ARROW-3192](https://issues.apache.org/jira/browse/ARROW-3192). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Read java/vector/src/main/java/org/apache/arrow/vector/ipc/message/MessageSerializer.java at the referenced lines 569 and 290 to understand the current ReadChannel allocation path. Investigate the proposed ArrowBufReadChannel abstraction and alternate MessageSerializer, with zero-copy reads for in-memory ArrowBuf payloads and allocation-copy fallback as the completion criteria.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100