apache / apache/hudi

[Vector Index] CREATE INDEX ... USING vector_index DDL + index definition wiring

Open
#18,855 0 comments 0 reactions 0 assignees View on GitHub
type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Part of #18676. RFC-104 / [design PR](https://github.com/chrevanthreddy/hudi/pull/1).

## Scope

User-facing entry point that triggers the bootstrap pipeline from sub-issue 5.

## Tasks

- Extend `hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/IndexCommands.scala` to recognize `vector_index` index type.
- Extend `hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/HoodieSparkIndexClient.java` to:
- Accept SQL options: `vectorColumn` (required), `numClusters` (optional, default from config), `fgPerCluster` (optional, default from config).
- Validate the column exists on the table and is of type `array` or `array`.
- Persist user-supplied params into `HoodieIndexDefinition` (so the bootstrap can read them back).
- Invoke `ScheduleIndexActionExecutor` → metadata writer bootstrap path.

Example DDL the change must support:

```sql
CREATE INDEX my_vec_idx ON hudi_tbl
USING vector_index (embedding)
OPTIONS (numClusters = '128', fgPerCluster = '2');
```

## Tests

- Negative test: missing `vectorColumn` option → clear error.
- Negative test: non-array column → clear error.
- Positive test: valid DDL parses and persists the `HoodieIndexDefinition` correctly.

## Depends on

- Sub-issues 1, 5 (need partition type + bootstrap implementation)

## Out of scope

`DROP INDEX` and `REFRESH INDEX` for vector indexes (later milestone).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in IndexCommands.scala to trace how index types are parsed, then inspect HoodieSparkIndexClient.java and the existing index-definition and scheduling paths. Add the vector_index DDL flow with the named options and validation, and cover missing vectorColumn, non-array columns, and valid definition persistence with tests for the stated cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, scala, spark
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.