Buffers allocated by ParquetFileReader.readVectored() are not being released
- Vorherrschende Sprache
- Java
- Sterne
- 3.1k
- Forks
- 1.6k
- Ø Merge
- 3 T. 12 Std.
- Gemergte PRs (30 T.)
- 33
Beschreibung
### Describe the bug, including details regarding any error messages, version, and platform.
this surfaces when running `TestParquetReader` with hadoop >= 3.4.0
```
mvn -Dtest=TestParquetReader -Dhadoop.version=3.4.1
```
This is disappointing as it means
1. the hadoop release process hasn't been doing the parquet build and test (it's an optional action)
2. this got through our own large scale QE testing.
The good news: this hasn't surfaced in production, presumably because nothing is trying to read into direct buffers -something parquet blocks automatically.
```
[ERROR] org.apache.parquet.hadoop.TestParquetReader.testCurrentRowIndex[2] -- Time elapsed: 0.013 s <<< ERROR!
org.apache.parquet.bytes.TrackingByteBufferAllocator$LeakedByteBufferException: 2 ByteBuffer object(s) is/are remained unreleased after closing this allocator.
at org.apache.parquet.bytes.TrackingByteBufferAllocator.close(TrackingByteBufferAllocator.java:171)
at org.apache.parquet.hadoop.TestParquetReader.closeAllocator(TestParquetReader.java:175)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
Caused by: org.apache.parquet.bytes.TrackingByteBufferAllocator$ByteBufferAllocationStacktraceException: Allocation stacktrace of the first ByteBuffer:
at org.apache.parquet.bytes.TrackingByteBufferAllocator$ByteBufferAllocationStacktraceException.create(TrackingByteBufferAllocator.java:100)
at org.apache.parquet.bytes.TrackingByteBufferAllocator$ByteBufferAllocationStacktraceException.access$300(TrackingByteBufferAllocator.java:94)
at org.apache.parquet.bytes.TrackingByteBufferAllocator.allocate(TrackingByteBufferAllocator.java:140)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.readVectored(RawLocalFileSystem.java:336)
at org.apache.hadoop.fs.BufferedFSInputStream.readVectored(BufferedFSInputStream.java:182)
at org.apache.hadoop.fs.FSDataInputStream.readVectored(FSDataInputStream.java:307)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.readVectored(ChecksumFileSystem.java:444)
at org.apache.hadoop.fs.FSDataInputStream.readVectored(FSDataInputStream.java:307)
```
### Component(s)
Core
Beitragsleitfaden
Für dieses Repository ist kein Beitragsleitfaden indexiert
Rechercherichtung
Beginne mit TestParquetReader.java rund um closeAllocator und dem Stacktrace von readVectored, und untersuche dann den Pfad von ParquetFileReader.readVectored() sowie die Verwendung von TrackingByteBufferAllocator. Reproduziere das Problem mit `mvn -Dtest=TestParquetReader -Dhadoop.version=3.4.1`; fertig ist es, wenn der Test unter Hadoop 3.4.0 oder neuer ohne geleakte Buffer besteht.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- java
- Bereich
- data-engineering
- Issue-Typ
- Bug
- Schwierigkeit
- 3/5
- Geschätzter Aufwand
- 1-2 Tage
- Aktivitätsstatus
- Veraltet
- Klarheit
- Größtenteils klar
- Anfängerfreundlichkeit
- 35/100