apache / apache/parquet-java

Off heap memory leaks with large binary fields using Snappy

未关闭
#2,114 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
Component: Java Component: Parquet Priority: Major Type: bug
主要语言
Java
星标
3.1k
派生
1.6k
平均合并
3 天 12 小时
30 天内合并 PR
33

描述

When I write a large pages (~100MB) that contains large binary fields (~1MB), the java application uses an unexpected amount of off-heap memory (1.2GB)

This problem was identified when using the `AvroParquetWriter` but its source lies in the parquet-hadoop submodule.

Diving a little bit deeper shows the following:
- writing fields into the ParquetWriter creates a SequenceBytesIn which is actually just a list of `BytesInput` for each field. When calling `bytes.writeAllTo(cos)` in the `CodecFactory`, it actually writes one `ByteInput` (which contains a single field) at a time.
- the `SnappyCompressor` receives the data in `setInput` one large field at a time. This calls `ByteBuffer.allocateDirect` each time with a growing size. But as the memory is actually allocated off-heap, this does not trigger the garbage collector which only sees small objects on the heap. The actual memory associated with the object is the size of all the fields added to the page until then, so off-heap the memory is growing quadratically.

I did not attach a pull request to this issue because I see multiple mitigation to the issue but I'm not really delighted by any of them:
- merge all the fields into one byte array before pushing them down to the `SnappyCompressor`. For instance we could replace the previous statement in the `CodecFactory` with `BytesInput.from(bytes.toByteArray()).writeAllTo(cos)`. But this generates an extra on-heap allocation the size of the whole page.
- force the `DirectBuffer` to be cleaned up with something like `((DirectBuffer)inputBuffer).cleaner().clean()` after having copied it to the new bigger buffer. The issue here would be that `DirectBuffer` is part of the internal API and is likely to be moved. Using reflexion could make the solution more resilient but is even "hackier" IMHO.

**Reporter**: [Remi Dettai](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=remi.dettai)

**Note**: *This issue was originally created as [PARQUET-1188](https://issues.apache.org/jira/browse/PARQUET-1188). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

贡献指南

这个仓库没有索引到贡献指南

调研方向

从 parquet-hadoop 子模块开始,跟踪 ParquetWriter 和 SequenceBytesIn 到 CodecFactory 和 SnappyCompressor。使用包含 1MB 二进制字段、大小约为 100MB 的页面重现该问题,然后评估提出的缓解路径。完成的标准是:大规模写入不再导致 off-heap 内存呈二次增长,同时保持 writer 的行为不变。

由索引模型根据 Issue 内容生成。

评估

技术栈
java
领域
performance
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
需要澄清
新手友好度
35/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。