apache / apache/hudi

[SUPPORT] Rollback failed clustering 0.12.2

Open
#10,964 6 comments 0 reactions 0 assignees View on GitHub
area:table-service
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**_Tips before filing an issue_**

- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?

- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.

**Describe the problem you faced**

Hello, this is a followup from https://github.com/apache/hudi/issues/10878. We managed to run clustering, but I'm obsessed with a potential recovery plan.
So what behavior I know - when `.commit.requested` and `.commit.inflight` created, but not `.commit` - then subsequent write will do a rollback. - this works for normal commits.
However, if I start clustering - if the job stops before `.inflight` is created - subsequent write will fail if affects partition present in `.replacecommit.requested` - controlled by [hoodie.clustering.updates.strategy](https://hudi.apache.org/docs/configurations/#hoodieclusteringupdatesstrategy). So here I can only either run clustering from CLI or just delete instant(can you confirm? per code looks like it's safe if there is no `.inflight`).
But - if it fails after start writing files(after `.replacecommit.inflight` is created, but before `.replacecommit` is created) - which choices do I have? As I checked through the code - it looks like there is no automatic rollback for `replacecommit`, and `hudi-cli` has rollback only for finished instants.
Given this, can you answer 2 questions:
1. If clustering failed after `.replacecommit.requested`, but before `.replacecommit.inflight` - is it safe to just delete commit file itself? Recently you added this PR and it looks to be doing exactly this https://github.com/apache/hudi/pull/10645/files
2. If clustering failed after `.replacecommit.inflight`, but before `.replacecommit` - what are the recovery steps? If I understand correctly there is no automatic rollback for it, but i may be wrong(working on reproduction).

**To Reproduce**

Steps to reproduce the behavior:

1.
2.
3.
4.

**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment Description**

* Hudi version : 0.12.2

* Spark version : 3.3.0

* Hive version :

* Hadoop version :

* Storage (HDFS/S3/GCS..) :

* Running on Docker? (yes/no) :

**Additional context**

Add any other context about the problem here.

**Stacktrace**

```Add the stacktrace of the error.```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the clustering and rollback code paths referenced in the issue, including hudi-cli and PR #10645. Trace the states around .replacecommit.requested, .replacecommit.inflight, and .replacecommit, then use the described failure scenarios as the reproduction basis. Done means both recovery cases are confirmed and the correct operational steps or required fix are documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.