apache / apache/hudi

When data table commits are out of order, occasionally mdt clean could fail due to reattempt using same instant time

Open
#18,236 0 comments 0 reactions 0 assignees View on GitHub
type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Bug Description

**What happened:**
When data table commits are out of order, occasionally mdt clean could fail due to reattempt using same instant time.

**What you expected:**
No failures when performing clean in mdt.

**Steps to reproduce:**
1. data table commits with c1, c2,c3... c10, c9.
2. ensure mdt clean happens just after c10 in data table commits. So that, mdt timeline will look like
dc1, dc2... dc8, dc10, t10_002.clean
3. after c9 is completed, clean will again be re-attempted in mdt. but it will attempt with t10_002.clean.
and we can notice the clean fails.

### Environment

**Hudi version:** 0.14.x and upwards
**Query engine:** (Spark/Flink/Trino etc) N/A
**Relevant configs:**

### Logs and Stack Trace

_No response_

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the described out-of-order commit sequence, ensuring mdt clean runs immediately after c10 and again after c9 completes. Trace why the second attempt reuses t10_002.clean, then verify that repeated clean attempts no longer fail for this timeline.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.