apache / apache/hudi

Rollback inflight indexing does not clean table config inflight MDT partition

Open
#17,169 1 comment 0 reactions 0 assignees View on GitHub
from-jira priority:high type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

steps:

latest hudi master branch (as of aug 24th)

create index on table inflight and writer crash

run cleaner table service. instants are rolled back, but inflight MDT partition in table config is not updated accordingly.

Steps to reproduce:
- Start async index building for a decent sized data table. (say rli). We can use HoodieIndexer for this purpose.
- Crash the index building process mid-way. Atleast we should have the requested and inllight in the timeline. or atelast the requested.
- hoodie.properties will have a reference to the "rli" as part of hoodie.properties (hoodie.table.metadata.partitions.inflight).
- Run HoodieIndexer to drop the RLI index which failed mid-way.
- hoodie.properties missed to clean up the entry for "hoodie.table.metadata.partitions.inflight".

Expected behavior:
once rollback or dropping of index command completes, we should not see the rli as part of "hoodie.table.metadata.partitions.inflight" in hoodie.properties

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-9740
- Type: Bug
- Fix version(s):
- 1.2.0

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the failure with HoodieIndexer by starting an asynchronous index build, crashing it, and running the cleaner or index-drop operation. Trace the rollback and table-config update paths that manage hoodie.table.metadata.partitions.inflight, then verify that the failed rli entry is absent from hoodie.properties after completion.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.