Off heap memory leaks with large binary fields using Snappy
- 主要言語
- Java
- スター
- 3.1k
- フォーク
- 1.6k
- 平均マージ
- 3日 12時間
- マージ済み PR(30日)
- 33
説明
When I write a large pages (~100MB) that contains large binary fields (~1MB), the java application uses an unexpected amount of off-heap memory (1.2GB)
This problem was identified when using the `AvroParquetWriter` but its source lies in the parquet-hadoop submodule.
Diving a little bit deeper shows the following:
- writing fields into the ParquetWriter creates a SequenceBytesIn which is actually just a list of `BytesInput` for each field. When calling `bytes.writeAllTo(cos)` in the `CodecFactory`, it actually writes one `ByteInput` (which contains a single field) at a time.
- the `SnappyCompressor` receives the data in `setInput` one large field at a time. This calls `ByteBuffer.allocateDirect` each time with a growing size. But as the memory is actually allocated off-heap, this does not trigger the garbage collector which only sees small objects on the heap. The actual memory associated with the object is the size of all the fields added to the page until then, so off-heap the memory is growing quadratically.
I did not attach a pull request to this issue because I see multiple mitigation to the issue but I'm not really delighted by any of them:
- merge all the fields into one byte array before pushing them down to the `SnappyCompressor`. For instance we could replace the previous statement in the `CodecFactory` with `BytesInput.from(bytes.toByteArray()).writeAllTo(cos)`. But this generates an extra on-heap allocation the size of the whole page.
- force the `DirectBuffer` to be cleaned up with something like `((DirectBuffer)inputBuffer).cleaner().clean()` after having copied it to the new bigger buffer. The issue here would be that `DirectBuffer` is part of the internal API and is likely to be moved. Using reflexion could make the solution more resilient but is even "hackier" IMHO.
**Reporter**: [Remi Dettai](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=remi.dettai)
**Note**: *This issue was originally created as [PARQUET-1188](https://issues.apache.org/jira/browse/PARQUET-1188). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
コントリビューションガイド
このリポジトリのコントリビューションガイドは索引されていません
調査の方向性
parquet-hadoop サブモジュールから開始し、ParquetWriter と SequenceBytesIn をたどって CodecFactory と SnappyCompressor まで追跡します。1MB のバイナリフィールドを含む約 100MB のページで問題を再現し、その後、提案されている緩和策の方向性を評価します。大規模な書き込みによって off-heap メモリが二次的に増加しなくなり、writer の動作が維持されれば完了です。
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- java
- 領域
- performance
- issue の種類
- バグ
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 活発さ
- 停滞
- 明瞭さ
- 説明が足りない
- 初心者へのやさしさ
- 35/100