[Vector Index] KMeans training stage using Spark MLlib
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Part of #18676. RFC-104 / [design PR](https://github.com/chrevanthreddy/hudi/pull/1).
## Scope
The IVF training step: produce K cluster centroids over the vector column of the data table. No MDT writes in this PR — pure training utility.
## Tasks
- New class `org.apache.hudi.index.vector.SparkVectorIndexTrainer` in `hudi-client/hudi-spark-client`.
- Inputs: dataset path (or `Dataset`), vector column name, `numClusters`, `trainingSampleSize`.
- Implementation:
- Reads the data table via Spark (parquet base files for now).
- Samples up to `trainingSampleSize` vectors (`Dataset.sample(...)` with deterministic seed for reproducibility).
- Converts the `array` vector column to `org.apache.spark.ml.linalg.Vector` via `VectorAssembler` / UDF.
- Fits `org.apache.spark.ml.clustering.KMeans` with configured `k`.
- Returns `Dataset` (`clusterId: int, centroid: array`).
## Tests
- Unit test on synthetic dataset (e.g. 3 well-separated Gaussian blobs in R^16): assert `numClusters` centroids returned and KMeans inertia is materially lower than a random-centroid baseline.
- Determinism test: same seed → same centroids.
## Depends on
- None (pure utility, doesn't touch MDT)
## Out of scope
Bootstrap orchestration, MDT writing, centroid persistence (deferred to a later sub-task).
Contributor guide
No contributing guide indexed for this repository
Research direction
Read RFC-104 and the linked design PR first, then inspect the hudi-client/hudi-spark-client module and the planned SparkVectorIndexTrainer entry point. Use a synthetic R^16 dataset with three separated Gaussian blobs to verify the requested centroid count and inertia improvement, and repeat training with the same seed to confirm deterministic centroids.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, machine-learning
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100