apache / apache/hudi

[Vector Index] Register vector_index_* MDT partition type + config knobs

Open
#18,850 0 comments 0 reactions 1 assignee Claimed by @rahil-c View on GitHub
type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Part of #18676. RFC-104 / [design PR](https://github.com/chrevanthreddy/hudi/pull/1).

## Scope

Schema/registration plumbing only — no data flow yet. Lands the new MDT partition type so subsequent sub-tasks have something to write into.

## Tasks

- Add `VECTOR_INDEX` enum constant in `hudi-common/src/main/java/org/apache/hudi/metadata/MetadataPartitionType.java` with:
- `partitionPath = "vector_index_"` (multi-instance prefix, mirrors `secondary_index_` and `expr_index_`)
- new `recordType` value
- override `getPartitionPath(metaClient, indexName)` to suffix the user index name (see `SECONDARY_INDEX` lines ~234–243)
- Add `PARTITION_NAME_VECTOR_INDEX_PREFIX = "vector_index_"` in `hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java` near lines 210–212.
- Add config knobs in `hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java` (mirror `RECORD_INDEX_*` block, lines 229–370):

Require config:
- `hoodie.metadata.vector.index.enable` (default `false`)

Optional configs for now"
- `hoodie.metadata.vector.index.num.clusters` (default `256`)
- `hoodie.metadata.vector.index.file.group.count.per.cluster` (default `1` — lets a cluster span N file groups)
- `hoodie.metadata.vector.index.training.sample.size` (default `1_000_000` — caps KMeans training rows for large tables)

## Tests

- Unit test in `TestMetadataPartitionType` confirming partition path derivation for `vector_index_myidx`.

## Out of scope

Payload schema, file-group mapping, KMeans, write path — covered by follow-up sub-issues.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.