[to be discussed] Users should be able to configure HUDI to attempt lazy failed writes rollback and/or clean before the startCommit
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Task Description
**What needs to be done:**
Add new config hoodie.cleaner.prewrite.cleaner.policy controls whether to run a CLEAN and/or rollback of failed writes before starting a new ingestion commit - enough if multi-writer is enabled (meaning rollback failed writes policy is LAZY_
**Why this task is needed:**
- We have had cases where write ingestion jobs repeatedly fail. Because we use LAZY policy, files from these failed writes were not implicitly rolled back between these ingestion jobs. This leads to a buildup of files in DFS partition directory and increase unbounded. And can cause them to hit file quota limits. To mitigate we had to manually run clean jobs or temporarily set policy to EAGER
- We want to handle potential future cases where write jobs make keep on repeatedly failing after completing the commit but before attempting post-commit clean, which can result in the same impact as above. For example, if there are infra issues like spark jobs failing/timing out when the HUDI write reaches the post-commit phase.
We have added this functionality to our internal 0.x HUDI build to address these issues, we can upstream once we achieve consensus.
### Task Type
Code improvement/refactoring
### Related Issues
**Parent feature issue:** (if applicable )
**Related issues:**
NOTE: Use `Relationships` button to add parent/blocking issues after issue is created.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing the proposed hoodie.cleaner.prewrite.cleaner.policy configuration and the startCommit lifecycle, including how lazy failed-write rollback and cleaning currently behave. Done means the agreed behavior for CLEAN and rollback before a new ingestion commit is implemented and validated for multi-writer use.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100