Buffers allocated by ParquetFileReader.readVectored() are not being released
- 主要言語
- Java
- スター
- 3.1k
- フォーク
- 1.6k
- 平均マージ
- 3日 12時間
- マージ済み PR(30日)
- 33
説明
### Describe the bug, including details regarding any error messages, version, and platform.
this surfaces when running `TestParquetReader` with hadoop >= 3.4.0
```
mvn -Dtest=TestParquetReader -Dhadoop.version=3.4.1
```
This is disappointing as it means
1. the hadoop release process hasn't been doing the parquet build and test (it's an optional action)
2. this got through our own large scale QE testing.
The good news: this hasn't surfaced in production, presumably because nothing is trying to read into direct buffers -something parquet blocks automatically.
```
[ERROR] org.apache.parquet.hadoop.TestParquetReader.testCurrentRowIndex[2] -- Time elapsed: 0.013 s <<< ERROR!
org.apache.parquet.bytes.TrackingByteBufferAllocator$LeakedByteBufferException: 2 ByteBuffer object(s) is/are remained unreleased after closing this allocator.
at org.apache.parquet.bytes.TrackingByteBufferAllocator.close(TrackingByteBufferAllocator.java:171)
at org.apache.parquet.hadoop.TestParquetReader.closeAllocator(TestParquetReader.java:175)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
Caused by: org.apache.parquet.bytes.TrackingByteBufferAllocator$ByteBufferAllocationStacktraceException: Allocation stacktrace of the first ByteBuffer:
at org.apache.parquet.bytes.TrackingByteBufferAllocator$ByteBufferAllocationStacktraceException.create(TrackingByteBufferAllocator.java:100)
at org.apache.parquet.bytes.TrackingByteBufferAllocator$ByteBufferAllocationStacktraceException.access$300(TrackingByteBufferAllocator.java:94)
at org.apache.parquet.bytes.TrackingByteBufferAllocator.allocate(TrackingByteBufferAllocator.java:140)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.readVectored(RawLocalFileSystem.java:336)
at org.apache.hadoop.fs.BufferedFSInputStream.readVectored(BufferedFSInputStream.java:182)
at org.apache.hadoop.fs.FSDataInputStream.readVectored(FSDataInputStream.java:307)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.readVectored(ChecksumFileSystem.java:444)
at org.apache.hadoop.fs.FSDataInputStream.readVectored(FSDataInputStream.java:307)
```
### Component(s)
Core
コントリビューションガイド
このリポジトリのコントリビューションガイドは索引されていません
調査の方向性
TestParquetReader.java の closeAllocator 周辺と readVectored のスタックトレースから始め、次に ParquetFileReader.readVectored() のパスと TrackingByteBufferAllocator の使用箇所を調べます。`mvn -Dtest=TestParquetReader -Dhadoop.version=3.4.1` で再現します。Hadoop 3.4.0 以降でバッファーのリークなしにテストがパスすれば完了です。
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- java
- 領域
- data-engineering
- issue の種類
- バグ
- 難易度
- 3/5
- 見積もり時間
- 1〜2日
- 活発さ
- 停滞
- 明瞭さ
- おおむね明確
- 初心者へのやさしさ
- 35/100