[Vector Index] Add Avro schema and HoodieMetadataPayload for vector records
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Part of #18676. RFC-104 / [design PR](https://github.com/chrevanthreddy/hudi/pull/1).
## Scope
Wire the on-disk record shape for the initial milestone. Only the minimum payload (cluster ID + raw vector) — RaBitQ codes, scalars, base-table pointers come later.
## Tasks
- Add `HoodieVectorIndexInfo` Avro record in `hudi-common/src/main/avro/HoodieMetadata.avsc` after `SecondaryIndexMetadata` (line ~557):
- `clusterId` (int)
- `vector` (array of float)
- `recordKey` is already the MDT record key — no need to duplicate
- In `hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java`:
- Add `SCHEMA_FIELD_ID_VECTOR_INDEX` field constant
- Add a payload constructor `HoodieMetadataPayload(String key, HoodieVectorIndexInfo vectorMetadata)`
- Add factory `createVectorIndexRecord(recordKey, clusterId, vector)` — mirror `createSecondaryIndexRecord` and `createRecordIndexUpdate` (line ~650)
- Override `combineMetadataPayloads` on the new enum entry (latest-wins for now).
## Tests
- Unit test in `TestHoodieMetadataPayload` for round-trip Avro serialization of `HoodieVectorIndexInfo`.
## Depends on
- #PARENT (sub-issue 1 — partition type must be registered first)
## Out of scope
KMeans training, bootstrap orchestration, DDL.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.