apache / apache/hudi

[SUPPORT] Hudi should rollback commit when metasync fails

Open
#11,701 6 comments 0 reactions 0 assignees View on GitHub
type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

hudi 0.15.0
---

AFAIK currently when metasync (hive/glue...) fails, hudi commits the data:

1. [hudi commit](https://github.com/apache/hudi/blob/be0068065d6727e6354e601846a4cf4d5e6d4f53/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala#L984)
2. [metasync action](
https://github.com/apache/hudi/blob/be0068065d6727e6354e601846a4cf4d5e6d4f53/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala#L1017)
3. [return failure status](https://github.com/apache/hudi/blob/be0068065d6727e6354e601846a4cf4d5e6d4f53/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala#L1021)
4. [eg of insert into cmd](https://github.com/apache/hudi/blob/c80b5596c1de08dc25a096be663241abf5de1b6e/hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala#L104)

So right now the user is aware there is a failure. However the hudi table is commited.

### proposal
I propose just after a metasync failure, a rollback operation on the current instant is done.

### Scenario
Let's say we promoted a type (int -> string). However athena does not support it. The glue metasync should raise an error.

1. If the commit is rollback, then the metastore can stay in a corrupted state (table partially updated) in worst case. **=> The table can still be read from athena**
2. However if it is not rollback, both metastore and hudi table will be in corrupted state: hudi will be promoted as string, but metastore is not able to support it. Then user has to rollback manually. **-> the table cannot be read anymore from athena**

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala around the commit at line 984 and metasync handling at lines 1017-1021; inspect the caller in InsertIntoHoodieTableCommand.scala around line 104. Confirm how the current instant is identified, then verify that a metasync failure rolls it back and that the reported failure behavior remains correct.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.