apache / apache/hudi

[to be discussed] Users should be able to configure HUDI to attempt lazy failed writes rollback and/or clean before the startCommit

Open
#17,901 2 comments 0 reactions 0 assignees View on GitHub
type:devtask
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Task Description

**What needs to be done:**

Add new config hoodie.cleaner.prewrite.cleaner.policy controls whether to run a CLEAN and/or rollback of failed writes before starting a new ingestion commit - enough if multi-writer is enabled (meaning rollback failed writes policy is LAZY_

**Why this task is needed:**
- We have had cases where write ingestion jobs repeatedly fail. Because we use LAZY policy, files from these failed writes were not implicitly rolled back between these ingestion jobs. This leads to a buildup of files in DFS partition directory and increase unbounded. And can cause them to hit file quota limits. To mitigate we had to manually run clean jobs or temporarily set policy to EAGER
- We want to handle potential future cases where write jobs make keep on repeatedly failing after completing the commit but before attempting post-commit clean, which can result in the same impact as above. For example, if there are infra issues like spark jobs failing/timing out when the HUDI write reaches the post-commit phase.

We have added this functionality to our internal 0.x HUDI build to address these issues, we can upstream once we achieve consensus.

### Task Type

Code improvement/refactoring

### Related Issues

**Parent feature issue:** (if applicable )
**Related issues:**
NOTE: Use `Relationships` button to add parent/blocking issues after issue is created.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the proposed hoodie.cleaner.prewrite.cleaner.policy configuration and the startCommit lifecycle, including how lazy failed-write rollback and cleaning currently behave. Done means the agreed behavior for CLEAN and rollback before a new ingestion commit is implemented and validated for multi-writer use.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.