apache / apache/arrow-java

[Java] Implement "ArrowBufReadChannel" abstraction and alternate MessageSerializer that uses this

未关闭
#371 2 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
Type: enhancement
主要语言
Java
星标
94
派生
152
平均合并
3 天 16 小时
30 天内合并 PR
11

描述

The current MessageSerializer implementation is wasteful when used to read an IPC payload that is already in-memory in an `ArrowBuf`. In particular, reads out of a `ReadChannel` require memory allocation

- https://github.com/apache/arrow/blob/master/java/vector/src/main/java/org/apache/arrow/vector/ipc/message/MessageSerializer.java#L569

- https://github.com/apache/arrow/blob/master/java/vector/src/main/java/org/apache/arrow/vector/ipc/message/MessageSerializer.java#L290

In C++, we have abstracted memory allocation out of the IPC read path so that zero-copy is possible. I suggest that a similar mechanism can be developed for Java to improve deserialization performance for in-memory messages. The new interface would return `ArrowBuf` when performing reads, which could be zero-copy when possible, but when not the current strategy of allocate-copy could be used

**Reporter**: [Wes McKinney](https://issues.apache.org/jira/browse/ARROW-3192) / @wesm

**Note**: *This issue was originally created as [ARROW-3192](https://issues.apache.org/jira/browse/ARROW-3192). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

贡献指南

打开贡献指南

调研方向

阅读 java/vector/src/main/java/org/apache/arrow/vector/ipc/message/MessageSerializer.java 中引用的第 569 行和第 290 行,以了解当前的 ReadChannel 分配路径。研究建议的 ArrowBufReadChannel 抽象和备用 MessageSerializer,将对内存中 ArrowBuf payload 的 zero-copy 读取以及分配并复制的 fallback 作为完成标准。

由索引模型根据 Issue 内容生成。

评估

技术栈
java
领域
data-engineering
Issue 类型
功能
难度
5/5
预计耗时
一周以上
活跃度
停滞
描述清晰度
基本清楚
新手友好度
35/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。