apache / apache/hudi

[Vector Index] Cluster assignment and bootstrap orchestration for MDT writes

Open
#18,854 0 comments 0 reactions 0 assignees View on GitHub
type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Part of #18676. RFC-104 / [design PR](https://github.com/chrevanthreddy/hudi/pull/1).

## Scope

The glue layer that produces and commits the records to the new MDT partition — this is the meat of the milestone. Brings together sub-issues 1–4.

## Tasks

- In `hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metadata/SparkHoodieBackedTableMetadataWriter.java`, implement:

```java
Pair> initializeVectorIndexPartition(
HoodieIndexDefinition indexDef,
Lazy>> latestFileSlices)
```

Steps:
1. Read base files for `(recordKey, vectorColumn)` via Spark.
2. Call `SparkVectorIndexTrainer` (sub-issue 4) → centroids.
3. Broadcast centroids; map each row to its nearest centroid → `(recordKey, vector, clusterId)`.
4. Build MDT records via `HoodieMetadataPayload.createVectorIndexRecord` (sub-issue 2).
5. Return `(numClusters * fgPerCluster, HoodieData)`.

- In `hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java`:
- Add `initializeVectorIndexPartition(...)` abstract hook.
- Add `VECTOR_INDEX` switch case in `initializeFromFilesystem` (lines ~475–524) calling the abstract hook then `initializeFilegroupsAndCommit`.

- Stub the Flink / Java client implementations with `UnsupportedOperationException` (RFC is explicitly Spark-first for now).

## Tests

- Unit test with mocked `SparkVectorIndexTrainer` confirming the produced `HoodieData` has the expected count and cluster distribution.
- End-to-end coverage is in sub-issue 7.

## Depends on

- Sub-issues 1, 2, 3, 4

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with RFC-104 and the dependent sub-issues, then read initializeFromFilesystem in hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java and the Spark implementation in hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metadata/SparkHoodieBackedTableMetadataWriter.java. Use the existing metadata writer tests as a pattern. Done means Spark produces clustered MDT records with the expected count and distribution, the VECTOR_INDEX hook commits them, and Flink/Java clients fail explicitly as specified.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
backend, data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.