[Vector Index] Register vector_index_* MDT partition type + config knobs
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Part of #18676. RFC-104 / [design PR](https://github.com/chrevanthreddy/hudi/pull/1).
## Scope
Schema/registration plumbing only — no data flow yet. Lands the new MDT partition type so subsequent sub-tasks have something to write into.
## Tasks
- Add `VECTOR_INDEX` enum constant in `hudi-common/src/main/java/org/apache/hudi/metadata/MetadataPartitionType.java` with:
- `partitionPath = "vector_index_"` (multi-instance prefix, mirrors `secondary_index_` and `expr_index_`)
- new `recordType` value
- override `getPartitionPath(metaClient, indexName)` to suffix the user index name (see `SECONDARY_INDEX` lines ~234–243)
- Add `PARTITION_NAME_VECTOR_INDEX_PREFIX = "vector_index_"` in `hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java` near lines 210–212.
- Add config knobs in `hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java` (mirror `RECORD_INDEX_*` block, lines 229–370):
Require config:
- `hoodie.metadata.vector.index.enable` (default `false`)
Optional configs for now"
- `hoodie.metadata.vector.index.num.clusters` (default `256`)
- `hoodie.metadata.vector.index.file.group.count.per.cluster` (default `1` — lets a cluster span N file groups)
- `hoodie.metadata.vector.index.training.sample.size` (default `1_000_000` — caps KMeans training rows for large tables)
## Tests
- Unit test in `TestMetadataPartitionType` confirming partition path derivation for `vector_index_myidx`.
## Out of scope
Payload schema, file-group mapping, KMeans, write path — covered by follow-up sub-issues.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.