apache / apache/hudi

Rollback left behind files from failed commit

Open
#14,200 4 comments 0 reactions 0 assignees View on GitHub
engine:flink type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Bug Description

**What happened:**

In Hudi-Flink, we observed that when timeline based marker servers were used, the rollback didn't delete all the files that were part of the failed commit. This causes, Hudi to include those files after archival of commits (.inflight && .requested).

**What you expected:**
100% of files created by ingestion commit should be deleted.

**Steps to reproduce:**
1. Simulate commit failure with flink restart. The ingestion should have generated marker files and partially wrote data files. Timeline will only contain .requested and .inflight
2. Next write of ingestion, will do a clean-up of failed commits. The clean-up finishes without errors but not all data files in the marker directory are deleted. This was evident from the logs in `MarkerBasedRollbackStrategy.getRollbackRequests` count of files read!=count files written during the commit time. We suspect a bug in timeline server contributing to this.

### Environment

**Hudi version:**
0.14
**Query engine:** (Spark/Flink/Trino etc)
Flink
**Relevant configs:**
hoodie.cleaner.prewrite.cleaner.policy=rollback_failed_writes
hoodie.write.markers.type=TIMELINE_SERVER_BASED

### Logs and Stack Trace

_No response_

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the failed Flink commit with timeline-based markers and inspect MarkerBasedRollbackStrategy.getRollbackRequests, focusing on the mismatch between files read and files written. Trace the timeline server's marker handling during cleanup. Done means rollback_failed_writes removes every data file created by the failed ingestion commit.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.