apache / apache/hudi

Make behavior of keeping File Writers open configurable

Open
#15,627 0 comments 0 reactions 1 assignee Claimed by @yihua View on GitHub
engine:spark from-jira priority:critical type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Currently, when writing in Spark we will be keeping the File Writers for individual partitions open as long as we're processing the batch which entails that all of the data written out will be kept in memory (at least the last row-group in case of Parquet writers) until batch is fully processed and all of the writers are closed.

While this allows us to better control how many files are created in every partition (we keep the writer open and hence we don't need to create a new file when a new record comes in), this brings a penalty of keeping all of the data in memory potentially leading to OOMs, longer GC cycles, etc

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-5385
- Type: Bug
- Epic: https://issues.apache.org/jira/browse/HUDI-3249
- Affects version(s):
- 0.12.1
- Fix version(s):
- 1.1.0

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.