[Feature] Support non-atomic hive writer
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.
### Motivation
Now that hive wtriter is atomic, it only makes a commit once after the job is completed. When users use hive writer to migrate historical data from hive partition table to paimon partition table, many small files may be generated. The number of small files is approximately equal to the number of map tasks. Because a map task may contain data of all partitions, the data size of a map task is about 128 MB. After being allocated to all partitions, the data files of each partition are relatively small. This will lead to a query for job oom such as #1253
### Solution
It is better to do non-atomic writes, commit each map task once, so that multiple map task commits trigger compact and reduce the number of small files.So we can add a parameter like **hive-write-atomic**. The default value is false, the user can specify true if atomic writing is required
### Anything else?
_No response_
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the Hive writer's commit path and how map-task output is handled. The proposed change is a hive-write-atomic parameter, defaulting to non-atomic commits while allowing atomic writing when enabled; done means both modes work without increasing small-file-related failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100