Add MDT row families and payload schema for vector indexes
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Parent: #19094
Related: #18676
PR: #19317
## Summary
Add the persisted MDT row families, keys, Avro models, payload builders, and serde contract for RFC-109 vector indexes.
This issue owns format-sensitive changes and must reflect the consistency model settled during RFC review before the schema lands.
## Scope
- Manifest, centroids, quantizer, generation, cluster, posting block/delta, tombstone, statistics, and source-instant marker rows.
- Avro union/model updates.
- Binary metadata keys and prefix/range helpers.
- Payload builders, merge behavior, and serde.
- Singleton active-generation pointer and per-generation manifest rows.
- Manifest bootstrap baseline and verified contiguous frontier/watermark.
- Cache-only `routingVersion` for epoch-free routing invalidation.
- Agreed bootstrap-format fields, including factor format/version, shard geometry, file-group count, and manifest kappa where applicable.
## Required correction before merge
- Remove `centroidEpoch` from all persisted schemas and generated APIs.
- Remove epoch bytes from centroid keys and key-range helpers.
- Do not add a replacement persisted epoch dimension.
- Add the source-instant marker family to the payload union and key namespace.
- Add a singleton active-generation pointer that can be flipped atomically during activation.
- Add the bootstrap baseline and verified contiguous frontier/watermark to each generation manifest.
- Model LIRE publication as one completed MDT delta commit; `routingVersion` invalidates caches but is not a query consistency boundary.
## Acceptance criteria
- Schema contains no centroid epoch field.
- Centroid keys are scoped by generation/chunk without epoch bytes.
- Marker keys uniquely identify index/generation/source instant as required by the accepted freshness contract.
- Manifest can represent the singleton reader-visible generation pointer plus generation state, bootstrap baseline, verified contiguous frontier, and routing cache version.
- Payload union, builders, combination semantics, and binary key helpers have round-trip tests.
- Prefix/range tests prove families do not overlap.
- Schema changes remain compatible with the planned rebootstrap boundary; no unnecessary migration path is introduced for an unshipped format.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with RFC-109, parent issue #19094, and the consistency model from the RFC review. Inspect the Avro models, payload union and builders, binary key helpers, and serde contract described in the scope. Done means the persisted schemas and keys satisfy the required corrections, with round-trip, prefix/range, and combination tests covering the acceptance criteria.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100