apache / apache/hudi

Support Rollback for Dropped Parititions

Open
#15,714 0 comments 0 reactions 0 assignees View on GitHub
from-jira priority:high type:devtask
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Currently, when a user tries to drop a partition using spark sql [https://spark.apache.org/docs/latest/sql-ref-syntax-ddl-alter-table.html#drop-partition] , and then perform a rollback on this dropped partition, they do not see this partition present when running  *SHOW PARTITIONS* command. The reason is that as part of drop partition operation, Hudi also deletes the partition from table metadata. However, rolling it back does not add the partition back to Hudi table metadata. Hence, *SHOW PARTITIONS* does not return the rolled back partition.

 

As part of drop partition command, Hudi will schedule a clean operation of this partition data treating this a HARD delete. However, it is possible that user rollsback the drop partition commit by the time the cleaner is run (or may be user turns off the cleaner). In such scenarios, even though the data is rolled back, the partition still does not appear in the table metadata leaving the Hudi table in a corrupt state.

 

We think we can enhance this functionality to support rollback for drop partitions. If we decide against it, then we should disallow rolling back of commits that drop partition so users don't end up in this state.

 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-5603
- Type: Task

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the Spark SQL DROP PARTITION, rollback, and SHOW PARTITIONS sequence described in the issue, including the cleaner timing or disabled-cleaner case. Trace how the drop removes Hudi table metadata and how rollback restores data. Done means rollback restores the partition in table metadata, or the affected rollback is explicitly disallowed to prevent corruption.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.