apache / apache/arrow-java

[Java] Memory leak when use VectorSchemaSlice slice

未关闭
#144 4 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Java
星标
94
派生
152
平均合并
3 天 16 小时
30 天内合并 PR
11

描述

### Describe the usage question you have. Please include as many useful details as possible.

In my Flink process function, I receive serialized VectorSchemaRoot data which needs to be deserialized for further processing. As I need to operate on it row by row, I utilized the slice function. However, this approach can lead to an increase in direct memory usage which in turn reduces the heap memory space in Flink. Ultimately, this can result in an Out Of Memory (OOM) exception as the heap memory space becomes insufficient. However, if I serialize the sliced VectorSchemaRoot and pass it as a parameter to the downstream function, which then deserializes it again, there will be no more OOM issues.

```
public void processElement(I value, ProcessFunction.Context context, Collector collector) throws Exception {
if (this.writerHelper == null) {
initWriterHelper();
}
reader = new ArrowStreamReader(new ByteArrayInputStream((byte[]) value), ArrowUtil.rootAllocator);
try {
while (reader.loadNextBatch()) {
VectorSchemaRoot vsr = reader.getVectorSchemaRoot();
int rowCount = vsr.getRowCount();
for (int i = 0; i < rowCount; i++) {
//split to row
VectorSchemaRoot row = vsr.slice(i, 1);
// this.writerHelper.write(row); This approach can result in a memory leak, whereas the following method will not.

ByteArrayOutputStream out = new ByteArrayOutputStream();
ArrowStreamWriter writer =
new ArrowStreamWriter(row, null, Channels.newChannel(out));
writer.start();
writer.writeBatch();
this.writerHelper.write(out.toByteArray());
row.clear();
row.close();
}
vsr.clear();
vsr.close();
}
} catch (Exception ex) {
ex.printStackTrace();
} finally {
reader.close();
}
}```

### Component(s)

Java

贡献指南

打开贡献指南

调研方向

从 processElement 示例开始,尤其关注 VectorSchemaRoot.slice(i, 1) 和 row.clear()/row.close() 调用。复现通过 writerHelper 写入切片行时的直接内存增长,然后将其与序列化再反序列化路径进行比较。完成标准是,切片行路径不再导致报告中的内存增长或 OOM。

由索引模型根据 Issue 内容生成。

评估

技术栈
java
领域
data
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
需要澄清
新手友好度
30/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。