[Vector Index] Cluster assignment and bootstrap orchestration for MDT writes
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Part of #18676. RFC-104 / [design PR](https://github.com/chrevanthreddy/hudi/pull/1).
## Scope
The glue layer that produces and commits the records to the new MDT partition — this is the meat of the milestone. Brings together sub-issues 1–4.
## Tasks
- In `hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metadata/SparkHoodieBackedTableMetadataWriter.java`, implement:
```java
Pair> initializeVectorIndexPartition(
HoodieIndexDefinition indexDef,
Lazy>> latestFileSlices)
```
Steps:
1. Read base files for `(recordKey, vectorColumn)` via Spark.
2. Call `SparkVectorIndexTrainer` (sub-issue 4) → centroids.
3. Broadcast centroids; map each row to its nearest centroid → `(recordKey, vector, clusterId)`.
4. Build MDT records via `HoodieMetadataPayload.createVectorIndexRecord` (sub-issue 2).
5. Return `(numClusters * fgPerCluster, HoodieData)`.
- In `hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java`:
- Add `initializeVectorIndexPartition(...)` abstract hook.
- Add `VECTOR_INDEX` switch case in `initializeFromFilesystem` (lines ~475–524) calling the abstract hook then `initializeFilegroupsAndCommit`.
- Stub the Flink / Java client implementations with `UnsupportedOperationException` (RFC is explicitly Spark-first for now).
## Tests
- Unit test with mocked `SparkVectorIndexTrainer` confirming the produced `HoodieData` has the expected count and cluster distribution.
- End-to-end coverage is in sub-issue 7.
## Depends on
- Sub-issues 1, 2, 3, 4
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with RFC-104 and the dependent sub-issues, then read initializeFromFilesystem in hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java and the Spark implementation in hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metadata/SparkHoodieBackedTableMetadataWriter.java. Use the existing metadata writer tests as a pattern. Done means Spark produces clustered MDT records with the expected count and distribution, the VECTOR_INDEX hook commits them, and Flink/Java clients fail explicitly as specified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- backend, data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100