apache / apache/hudi

[Vector Index] Add Avro schema and HoodieMetadataPayload for vector records

Open
#18,851 1 comment 0 reactions 1 assignee Claimed by @rahil-c View on GitHub
type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Part of #18676. RFC-104 / [design PR](https://github.com/chrevanthreddy/hudi/pull/1).

## Scope

Wire the on-disk record shape for the initial milestone. Only the minimum payload (cluster ID + raw vector) — RaBitQ codes, scalars, base-table pointers come later.

## Tasks

- Add `HoodieVectorIndexInfo` Avro record in `hudi-common/src/main/avro/HoodieMetadata.avsc` after `SecondaryIndexMetadata` (line ~557):
- `clusterId` (int)
- `vector` (array of float)
- `recordKey` is already the MDT record key — no need to duplicate
- In `hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java`:
- Add `SCHEMA_FIELD_ID_VECTOR_INDEX` field constant
- Add a payload constructor `HoodieMetadataPayload(String key, HoodieVectorIndexInfo vectorMetadata)`
- Add factory `createVectorIndexRecord(recordKey, clusterId, vector)` — mirror `createSecondaryIndexRecord` and `createRecordIndexUpdate` (line ~650)
- Override `combineMetadataPayloads` on the new enum entry (latest-wins for now).

## Tests

- Unit test in `TestHoodieMetadataPayload` for round-trip Avro serialization of `HoodieVectorIndexInfo`.

## Depends on

- #PARENT (sub-issue 1 — partition type must be registered first)

## Out of scope

KMeans training, bootstrap orchestration, DDL.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.