Rollback left behind files from failed commit
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Bug Description
**What happened:**
In Hudi-Flink, we observed that when timeline based marker servers were used, the rollback didn't delete all the files that were part of the failed commit. This causes, Hudi to include those files after archival of commits (.inflight && .requested).
**What you expected:**
100% of files created by ingestion commit should be deleted.
**Steps to reproduce:**
1. Simulate commit failure with flink restart. The ingestion should have generated marker files and partially wrote data files. Timeline will only contain .requested and .inflight
2. Next write of ingestion, will do a clean-up of failed commits. The clean-up finishes without errors but not all data files in the marker directory are deleted. This was evident from the logs in `MarkerBasedRollbackStrategy.getRollbackRequests` count of files read!=count files written during the commit time. We suspect a bug in timeline server contributing to this.
### Environment
**Hudi version:**
0.14
**Query engine:** (Spark/Flink/Trino etc)
Flink
**Relevant configs:**
hoodie.cleaner.prewrite.cleaner.policy=rollback_failed_writes
hoodie.write.markers.type=TIMELINE_SERVER_BASED
### Logs and Stack Trace
_No response_
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the failed Flink commit with timeline-based markers and inspect MarkerBasedRollbackStrategy.getRollbackRequests, focusing on the mismatch between files read and files written. Trace the timeline server's marker handling during cleanup. Done means rollback_failed_writes removes every data file created by the failed ingestion commit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, distributed-systems, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100