apache / apache/parquet-java

stream through files when writing the _metadata file to reduce memory usage

Open
#1,571 0 comments 0 reactions 0 assignees View on GitHub
Component: Java Component: Parquet Priority: Major Type: enhancement
Dominant language
Java
Stars
3.1k
Forks
1.6k
Avg merge
3d 12h
Merged PRs (30d)
33

Description

Currently we load all the footers of the part files in memory in the committer to write the _metadata file.
We could open the _metadata file first and then add the metadata for each part file one by one. That would reduce the required memory.

**Reporter**: [Julien Le Dem](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=julienledem) / @julienledem

**Note**: *This issue was originally created as [PARQUET-91](https://issues.apache.org/jira/browse/PARQUET-91). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.