[Vector Index] CREATE INDEX ... USING vector_index DDL + index definition wiring
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Part of #18676. RFC-104 / [design PR](https://github.com/chrevanthreddy/hudi/pull/1).
## Scope
User-facing entry point that triggers the bootstrap pipeline from sub-issue 5.
## Tasks
- Extend `hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/IndexCommands.scala` to recognize `vector_index` index type.
- Extend `hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/HoodieSparkIndexClient.java` to:
- Accept SQL options: `vectorColumn` (required), `numClusters` (optional, default from config), `fgPerCluster` (optional, default from config).
- Validate the column exists on the table and is of type `array` or `array`.
- Persist user-supplied params into `HoodieIndexDefinition` (so the bootstrap can read them back).
- Invoke `ScheduleIndexActionExecutor` → metadata writer bootstrap path.
Example DDL the change must support:
```sql
CREATE INDEX my_vec_idx ON hudi_tbl
USING vector_index (embedding)
OPTIONS (numClusters = '128', fgPerCluster = '2');
```
## Tests
- Negative test: missing `vectorColumn` option → clear error.
- Negative test: non-array column → clear error.
- Positive test: valid DDL parses and persists the `HoodieIndexDefinition` correctly.
## Depends on
- Sub-issues 1, 5 (need partition type + bootstrap implementation)
## Out of scope
`DROP INDEX` and `REFRESH INDEX` for vector indexes (later milestone).
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in IndexCommands.scala to trace how index types are parsed, then inspect HoodieSparkIndexClient.java and the existing index-definition and scheduling paths. Add the vector_index DDL flow with the named options and validation, and cover missing vectorColumn, non-array columns, and valid definition persistence with tests for the stated cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, scala, spark
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100