apache / apache/hudi

[Vector Index] KMeans training stage using Spark MLlib

Open
#18,853 0 comments 0 reactions 0 assignees View on GitHub
type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Part of #18676. RFC-104 / [design PR](https://github.com/chrevanthreddy/hudi/pull/1).

## Scope

The IVF training step: produce K cluster centroids over the vector column of the data table. No MDT writes in this PR — pure training utility.

## Tasks

- New class `org.apache.hudi.index.vector.SparkVectorIndexTrainer` in `hudi-client/hudi-spark-client`.
- Inputs: dataset path (or `Dataset`), vector column name, `numClusters`, `trainingSampleSize`.
- Implementation:
- Reads the data table via Spark (parquet base files for now).
- Samples up to `trainingSampleSize` vectors (`Dataset.sample(...)` with deterministic seed for reproducibility).
- Converts the `array` vector column to `org.apache.spark.ml.linalg.Vector` via `VectorAssembler` / UDF.
- Fits `org.apache.spark.ml.clustering.KMeans` with configured `k`.
- Returns `Dataset` (`clusterId: int, centroid: array`).

## Tests

- Unit test on synthetic dataset (e.g. 3 well-separated Gaussian blobs in R^16): assert `numClusters` centroids returned and KMeans inertia is materially lower than a random-centroid baseline.
- Determinism test: same seed → same centroids.

## Depends on

- None (pure utility, doesn't touch MDT)

## Out of scope

Bootstrap orchestration, MDT writing, centroid persistence (deferred to a later sub-task).

Contributor guide

No contributing guide indexed for this repository

Research direction

Read RFC-104 and the linked design PR first, then inspect the hudi-client/hudi-spark-client module and the planned SparkVectorIndexTrainer entry point. Use a synthetic R^16 dataset with three separated Gaussian blobs to verify the requested centroid count and inertia improvement, and repeat training with the same seed to confirm deterministic centroids.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, machine-learning
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.