parquet writer runs into OOM during writing
- 主要言語
- Java
- スター
- 3.1k
- フォーク
- 1.6k
- 平均マージ
- 3日 12時間
- マージ済み PR(30日)
- 33
説明
Hi team,
We are getting OOM error on trying to writer data to the parquet file. Please check below stack trace:
> Caused by: java.lang.OutOfMemoryError: Direct buffer memory at java.base/java.nio.Bits.reserveMemory(Bits.java:175) at java.base/java.nio.DirectByteBuffer.(DirectByteBuffer.java:118) at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) at org.apache.parquet.hadoop.codec.SnappyCompressor.setInput(SnappyCompressor.java:97) at org.apache.parquet.hadoop.codec.NonBlockedCompressorStream.write(NonBlockedCompressorStream.java:48) at org.apache.parquet.bytes.CapacityByteArrayOutputStream.writeToOutput(CapacityByteArrayOutputStream.java:227) at org.apache.parquet.bytes.CapacityByteArrayOutputStream.writeTo(CapacityByteArrayOutputStream.java:247) at org.apache.parquet.bytes.BytesInput$CapacityBAOSBytesInput.writeAllTo(BytesInput.java:405) at org.apache.parquet.bytes.BytesInput$SequenceBytesIn.writeAllTo(BytesInput.java:296) at org.apache.parquet.hadoop.CodecFactory$HeapBytesCompressor.compress(CodecFactory.java:164) at org.apache.parquet.hadoop.ColumnChunkPageWriteStore$ColumnChunkPageWriter.writePage(ColumnChunkPageWriteStore.java:95) at org.apache.parquet.column.impl.ColumnWriterV1.writePage(ColumnWriterV1.java:147) at org.apache.parquet.column.impl.ColumnWriterV1.flush(ColumnWriterV1.java:235) at org.apache.parquet.column.impl.ColumnWriteStoreV1.flush(ColumnWriteStoreV1.java:122) at org.apache.parquet.hadoop.InternalParquetRecordWriter.flushRowGroupToStore(InternalParquetRecordWriter.java:172) at org.apache.parquet.hadoop.InternalParquetRecordWriter.checkBlockSizeReached(InternalParquetRecordWriter.java:148) at org.apache.parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:130) at org.apache.parquet.hadoop.ParquetWriter.write(ParquetWriter.java:299) at com.fivetran.warehouses.common.parquet.AvroBasedParquetWriterAdapter.write(AvroBasedParquetWriterAdapter.java:39)
We believe that most of the memory is being consumed by slabs. From below warning we can see that a content column acquired 108 slabs:
> [content] optional binary content (UTF8) { r:0 d: RunLengthBitPackingHybrid 64 bytes data: FallbackValuesWriter{ data: initial: DictionaryValuesWriter{ data: initial: dict:0 data: initial: values:0 data: initial:} data: fallback: PLAIN CapacityByteArrayOutputStream 108 slabs, 162,188,576 bytes data:} pages: ColumnChunkPageWriter ConcatenatingByteArrayCollector 0 slabs, 0 bytes total: 162,188,590/162,188,640 }
Could you please help us resolve this issue?
Thanks
**Reporter**: [Ketki Bukkawar](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=ketki.bukkawar)
**Note**: *This issue was originally created as [PARQUET-2166](https://issues.apache.org/jira/browse/PARQUET-2166). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
コントリビューションガイド
このリポジトリのコントリビューションガイドは索引されていません
調査の方向性
Start with InternalParquetRecordWriter.checkBlockSizeReached and flushRowGroupToStore in the stack trace, then inspect ColumnChunkPageWriteStore and SnappyCompressor around the reported 162 MB content column and 108 slabs. Reproduce the direct-buffer OOM with a similarly large content column. Done means parquet writing completes without the reported OOM and the behavior is covered by an appropriate regression test.
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- java
- 領域
- data-engineering
- issue の種類
- バグ
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 活発さ
- 停滞
- 明瞭さ
- 説明が足りない
- 初心者へのやさしさ
- 25/100