Buffers allocated by ParquetFileReader.readVectored() are not being released
- Lenguaje dominante
- Java
- Estrellas
- 3.1k
- Forks
- 1.6k
- Merge medio
- 3 d 12 h
- PR fusionados (30 d)
- 33
Descripción
### Describe the bug, including details regarding any error messages, version, and platform.
this surfaces when running `TestParquetReader` with hadoop >= 3.4.0
```
mvn -Dtest=TestParquetReader -Dhadoop.version=3.4.1
```
This is disappointing as it means
1. the hadoop release process hasn't been doing the parquet build and test (it's an optional action)
2. this got through our own large scale QE testing.
The good news: this hasn't surfaced in production, presumably because nothing is trying to read into direct buffers -something parquet blocks automatically.
```
[ERROR] org.apache.parquet.hadoop.TestParquetReader.testCurrentRowIndex[2] -- Time elapsed: 0.013 s <<< ERROR!
org.apache.parquet.bytes.TrackingByteBufferAllocator$LeakedByteBufferException: 2 ByteBuffer object(s) is/are remained unreleased after closing this allocator.
at org.apache.parquet.bytes.TrackingByteBufferAllocator.close(TrackingByteBufferAllocator.java:171)
at org.apache.parquet.hadoop.TestParquetReader.closeAllocator(TestParquetReader.java:175)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
Caused by: org.apache.parquet.bytes.TrackingByteBufferAllocator$ByteBufferAllocationStacktraceException: Allocation stacktrace of the first ByteBuffer:
at org.apache.parquet.bytes.TrackingByteBufferAllocator$ByteBufferAllocationStacktraceException.create(TrackingByteBufferAllocator.java:100)
at org.apache.parquet.bytes.TrackingByteBufferAllocator$ByteBufferAllocationStacktraceException.access$300(TrackingByteBufferAllocator.java:94)
at org.apache.parquet.bytes.TrackingByteBufferAllocator.allocate(TrackingByteBufferAllocator.java:140)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.readVectored(RawLocalFileSystem.java:336)
at org.apache.hadoop.fs.BufferedFSInputStream.readVectored(BufferedFSInputStream.java:182)
at org.apache.hadoop.fs.FSDataInputStream.readVectored(FSDataInputStream.java:307)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.readVectored(ChecksumFileSystem.java:444)
at org.apache.hadoop.fs.FSDataInputStream.readVectored(FSDataInputStream.java:307)
```
### Component(s)
Core
Guía de contribución
No hay ninguna guía de contribución indexada para este repositorio
Línea de trabajo
Comienza con TestParquetReader.java alrededor de closeAllocator y el stack trace de readVectored; después inspecciona la ruta de ParquetFileReader.readVectored() y el uso de TrackingByteBufferAllocator. Reprodúcelo con `mvn -Dtest=TestParquetReader -Dhadoop.version=3.4.1`; se considera terminado cuando la prueba pasa sin buffers filtrados con Hadoop 3.4.0 o posterior.
Escrito por el modelo de indexación a partir del texto del issue.
Evaluación
- Stack tecnológico
- java
- Área
- data-engineering
- Tipo de issue
- Error
- Dificultad
- 3/5
- Tiempo estimado
- 1-2 días
- Estado de actividad
- Estancado
- Claridad
- Bastante claro
- Aptitud para principiantes
- 35/100