ColumnChunkPageWriter uses only heap memory.
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 1.6k
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 33
Description
After PARQUET-160 was resolved, ColumnChunkPageWriter started using ConcatenatingByteArrayCollector. There are all data is collected in the List of byte[], before writing the page. No way to use direct memory for allocating buffers. ByteBufferAllocator is present in the [ColumnChunkPageWriter](https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ColumnChunkPageWriteStore.java#L73) class, but never used.
Using of java heap space in some cases can cause OOM exceptions or GC's overhead.
ByteBufferAllocator should be used in the ConcatenatingByteArrayCollector or OutputStream classes.
**Reporter**: [Vitalii Diravka](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=vitalii) / @vdiravka
**Assignee**: [Vitalii Diravka](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=vitalii) / @vdiravka
#### Related issues:
- [Error: SYSTEM ERROR: RuntimeException: Unknown logical type ](https://issues.apache.org/jira/browse/DRILL-7825) (Is contained by)
- [Replace ParquetColumnChunkPageWriter with original Parquet class](https://issues.apache.org/jira/browse/DRILL-7906) (Is contained by)
- [Out of heap running CTAS against text delimited](https://issues.apache.org/jira/browse/DRILL-5544) (relates to)
- [Support configurable for DirectByteBufferAllocator from Hadoop Configuration](https://github.com/apache/parquet-java/issues/2443) (is related to)
- [Improvements in ByteBuffer read path](https://github.com/apache/parquet-java/issues/1534) (is related to)
- [Simplify CapacityByteArrayOutputStream](https://github.com/apache/parquet-java/issues/1712) (is related to)
**Note**: *This issue was originally created as [PARQUET-1006](https://issues.apache.org/jira/browse/PARQUET-1006). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.