Indexing action for one of several uninitialized secondary indexes marks the partition complete without building it
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**Describe the problem**
When a table has two or more secondary index definitions whose metadata partitions are not initialized (for example after the metadata table was deleted and is being rebuilt while `.hoodie/.index_defs/index.json` survived), an indexing action for one of them builds nothing, yet the action completes and the partition is added to `hoodie.table.metadata.partitions`.
**Why**
`HoodieBackedTableMetadataWriter.buildMetadataPartitions` reduces the plan's partition paths to partition types. `SecondaryIndexer.buildInitialization` (and `ExpressionIndexer`) then pick the partition from every uninitialized definition of the type via `getSecondaryIndexPartitionsToInit`, and when that is not exactly one they log "Skipping secondary index initialization as only one secondary index bootstrap at a time is supported" and return nothing. `RunIndexActionExecutor` still calls `updateTableConfigAndTimeline` and marks the requested partition complete.
**Impact**
An empty index partition that readers trust for data skipping, on every table with more than one secondary (or expression) index whose metadata table is rebuilt index by index.
**Reproduction**
Register two secondary index definitions on a table with an initialized metadata table, then run `HoodieIndexer` (or `SparkRDDWriteClient.scheduleIndexing` + `index`) for one of them. `hoodie.table.metadata.partitions` lists the index; `HoodieTableMetadataUtil.metadataPartitionExists` is false and the partition has no file slices. A regression test is in the fixing PR.
**Expected**
The indexing action initializes exactly the partition its plan names whenever its definition exists, regardless of how many other definitions are uninitialized; a regular write keeps inferring from the uninitialized definitions.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.