apache / apache/paimon

[Feature] Support non-atomic hive writer

Open
#1,260 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.

### Motivation

Now that hive wtriter is atomic, it only makes a commit once after the job is completed. When users use hive writer to migrate historical data from hive partition table to paimon partition table, many small files may be generated. The number of small files is approximately equal to the number of map tasks. Because a map task may contain data of all partitions, the data size of a map task is about 128 MB. After being allocated to all partitions, the data files of each partition are relatively small. This will lead to a query for job oom such as #1253

### Solution

It is better to do non-atomic writes, commit each map task once, so that multiple map task commits trigger compact and reduce the number of small files.So we can add a parameter like **hive-write-atomic**. The default value is false, the user can specify true if atomic writing is required

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the Hive writer's commit path and how map-task output is handled. The proposed change is a hive-write-atomic parameter, defaulting to non-atomic commits while allowing atomic writing when enabled; done means both modes work without increasing small-file-related failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.