Support Rollback for Dropped Parititions
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Currently, when a user tries to drop a partition using spark sql [https://spark.apache.org/docs/latest/sql-ref-syntax-ddl-alter-table.html#drop-partition] , and then perform a rollback on this dropped partition, they do not see this partition present when running *SHOW PARTITIONS* command. The reason is that as part of drop partition operation, Hudi also deletes the partition from table metadata. However, rolling it back does not add the partition back to Hudi table metadata. Hence, *SHOW PARTITIONS* does not return the rolled back partition.
As part of drop partition command, Hudi will schedule a clean operation of this partition data treating this a HARD delete. However, it is possible that user rollsback the drop partition commit by the time the cleaner is run (or may be user turns off the cleaner). In such scenarios, even though the data is rolled back, the partition still does not appear in the table metadata leaving the Hudi table in a corrupt state.
We think we can enhance this functionality to support rollback for drop partitions. If we decide against it, then we should disallow rolling back of commits that drop partition so users don't end up in this state.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5603
- Type: Task
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the Spark SQL DROP PARTITION, rollback, and SHOW PARTITIONS sequence described in the issue, including the cleaner timing or disabled-cleaner case. Trace how the drop removes Hudi table metadata and how rollback restores data. Done means rollback restores the partition in table metadata, or the affected rollback is explicitly disallowed to prevent corruption.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100