Rollback inflight indexing does not clean table config inflight MDT partition
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
steps:
latest hudi master branch (as of aug 24th)
create index on table inflight and writer crash
run cleaner table service. instants are rolled back, but inflight MDT partition in table config is not updated accordingly.
Steps to reproduce:
- Start async index building for a decent sized data table. (say rli). We can use HoodieIndexer for this purpose.
- Crash the index building process mid-way. Atleast we should have the requested and inllight in the timeline. or atelast the requested.
- hoodie.properties will have a reference to the "rli" as part of hoodie.properties (hoodie.table.metadata.partitions.inflight).
- Run HoodieIndexer to drop the RLI index which failed mid-way.
- hoodie.properties missed to clean up the entry for "hoodie.table.metadata.partitions.inflight".
Expected behavior:
once rollback or dropping of index command completes, we should not see the rli as part of "hoodie.table.metadata.partitions.inflight" in hoodie.properties
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-9740
- Type: Bug
- Fix version(s):
- 1.2.0
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the failure with HoodieIndexer by starting an asynchronous index build, crashing it, and running the cleaner or index-drop operation. Trace the rollback and table-config update paths that manage hoodie.table.metadata.partitions.inflight, then verify that the failed rli entry is absent from hoodie.properties after completion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100