apache / apache/hudi

Indexing action for one of several uninitialized secondary indexes marks the partition complete without building it

Open
#19,716 1 comment 0 reactions 1 assignee Claimed by @Davis-Zhang-Onehouse View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem**

When a table has two or more secondary index definitions whose metadata partitions are not initialized (for example after the metadata table was deleted and is being rebuilt while `.hoodie/.index_defs/index.json` survived), an indexing action for one of them builds nothing, yet the action completes and the partition is added to `hoodie.table.metadata.partitions`.

**Why**

`HoodieBackedTableMetadataWriter.buildMetadataPartitions` reduces the plan's partition paths to partition types. `SecondaryIndexer.buildInitialization` (and `ExpressionIndexer`) then pick the partition from every uninitialized definition of the type via `getSecondaryIndexPartitionsToInit`, and when that is not exactly one they log "Skipping secondary index initialization as only one secondary index bootstrap at a time is supported" and return nothing. `RunIndexActionExecutor` still calls `updateTableConfigAndTimeline` and marks the requested partition complete.

**Impact**

An empty index partition that readers trust for data skipping, on every table with more than one secondary (or expression) index whose metadata table is rebuilt index by index.

**Reproduction**

Register two secondary index definitions on a table with an initialized metadata table, then run `HoodieIndexer` (or `SparkRDDWriteClient.scheduleIndexing` + `index`) for one of them. `hoodie.table.metadata.partitions` lists the index; `HoodieTableMetadataUtil.metadataPartitionExists` is false and the partition has no file slices. A regression test is in the fixing PR.

**Expected**

The indexing action initializes exactly the partition its plan names whenever its definition exists, regardless of how many other definitions are uninitialized; a regular write keeps inferring from the uninitialized definitions.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.