ColumnChunkPageWriter uses only heap memory.
- Lenguaje dominante
- Java
- Estrellas
- 3.1k
- Forks
- 1.6k
- Merge medio
- 3 d 12 h
- PR fusionados (30 d)
- 33
Descripción
After PARQUET-160 was resolved, ColumnChunkPageWriter started using ConcatenatingByteArrayCollector. There are all data is collected in the List of byte[], before writing the page. No way to use direct memory for allocating buffers. ByteBufferAllocator is present in the [ColumnChunkPageWriter](https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ColumnChunkPageWriteStore.java#L73) class, but never used.
Using of java heap space in some cases can cause OOM exceptions or GC's overhead.
ByteBufferAllocator should be used in the ConcatenatingByteArrayCollector or OutputStream classes.
**Reporter**: [Vitalii Diravka](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=vitalii) / @vdiravka
**Assignee**: [Vitalii Diravka](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=vitalii) / @vdiravka
#### Related issues:
- [Error: SYSTEM ERROR: RuntimeException: Unknown logical type ](https://issues.apache.org/jira/browse/DRILL-7825) (Is contained by)
- [Replace ParquetColumnChunkPageWriter with original Parquet class](https://issues.apache.org/jira/browse/DRILL-7906) (Is contained by)
- [Out of heap running CTAS against text delimited](https://issues.apache.org/jira/browse/DRILL-5544) (relates to)
- [Support configurable for DirectByteBufferAllocator from Hadoop Configuration](https://github.com/apache/parquet-java/issues/2443) (is related to)
- [Improvements in ByteBuffer read path](https://github.com/apache/parquet-java/issues/1534) (is related to)
- [Simplify CapacityByteArrayOutputStream](https://github.com/apache/parquet-java/issues/1712) (is related to)
**Note**: *This issue was originally created as [PARQUET-1006](https://issues.apache.org/jira/browse/PARQUET-1006). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
Guía de contribución
No hay ninguna guía de contribución indexada para este repositorio
Línea de trabajo
Comienza en parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ColumnChunkPageWriteStore.java y sigue cómo ColumnChunkPageWriter pasa su ByteBufferAllocator a ConcatenatingByteArrayCollector o al OutputStream correspondiente. Verifica dónde se acumulan los datos de las páginas antes de la escritura y determina si la ruta completa realiza las asignaciones mediante el allocator proporcionado, en lugar de usar únicamente arrays de bytes del heap.
Escrito por el modelo de indexación a partir del texto del issue.
Evaluación
- Stack tecnológico
- java
- Área
- data-engineering
- Tipo de issue
- Error
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Estado de actividad
- Estancado
- Claridad
- Bastante claro
- Aptitud para principiantes
- 42/100