[Vector Index] End-to-end integration test for vector index bootstrap
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Part of #18676. RFC-104 / [design PR](https://github.com/chrevanthreddy/hudi/pull/1).
## Scope
Prove the milestone-1 pipeline works end-to-end on Spark.
## Tasks
- New Scala test `hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestVectorIndexBootstrap.scala`.
- Test flow:
1. Write a small Hudi MOR table (~1k rows) with a `vector` column populated by synthetic embeddings drawn from K well-separated Gaussian clusters in R^32.
2. Run `CREATE INDEX vec_idx ON tbl USING vector_index (vector) OPTIONS (numClusters = 'K', fgPerCluster = '2')`.
3. Assertions:
- MDT partition `vector_index_vec_idx` exists on disk.
- MDT file-group count equals `K * fgPerCluster`.
- Every base-table record key appears exactly once in the MDT partition.
- Each MDT record's `clusterId` is in `[0, K)` and its `vector` field matches the base-table vector for that key.
- Bonus assertion: KMeans recovered the synthetic clusters (centroid-to-truth nearest-neighbor distance below a threshold).
## Depends on
- Sub-issues 1–6 (this is the integration test that lights up the whole milestone)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with RFC-104, its design PR, and the dependent sub-issues, then create the named Scala test at hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestVectorIndexBootstrap.scala. Done means the Spark end-to-end flow verifies the MDT partition, expected file-group count, one matching MDT record per base-table key, valid cluster IDs, and matching vectors; the KMeans recovery check is optional.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data-engineering, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100