apache / apache/hudi

[Vector Index] End-to-end integration test for vector index bootstrap

Open
#18,856 0 comments 0 reactions 0 assignees View on GitHub
type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Part of #18676. RFC-104 / [design PR](https://github.com/chrevanthreddy/hudi/pull/1).

## Scope

Prove the milestone-1 pipeline works end-to-end on Spark.

## Tasks

- New Scala test `hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestVectorIndexBootstrap.scala`.
- Test flow:
1. Write a small Hudi MOR table (~1k rows) with a `vector` column populated by synthetic embeddings drawn from K well-separated Gaussian clusters in R^32.
2. Run `CREATE INDEX vec_idx ON tbl USING vector_index (vector) OPTIONS (numClusters = 'K', fgPerCluster = '2')`.
3. Assertions:
- MDT partition `vector_index_vec_idx` exists on disk.
- MDT file-group count equals `K * fgPerCluster`.
- Every base-table record key appears exactly once in the MDT partition.
- Each MDT record's `clusterId` is in `[0, K)` and its `vector` field matches the base-table vector for that key.
- Bonus assertion: KMeans recovered the synthetic clusters (centroid-to-truth nearest-neighbor distance below a threshold).

## Depends on

- Sub-issues 1–6 (this is the integration test that lights up the whole milestone)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with RFC-104, its design PR, and the dependent sub-issues, then create the named Scala test at hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestVectorIndexBootstrap.scala. Done means the Spark end-to-end flow verifies the MDT partition, expected file-group count, one matching MDT record per base-table key, valid cluster IDs, and matching vectors; the KMeans recovery check is optional.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data-engineering, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.