RFC-109: MDT-backed vector index umbrella tracking issue

# RFC-109: MDT-backed vector index umbrella tracking issue

## Summary

This issue is the implementation source of truth for RFC-109: native MDT-backed vector index support in Apache Hudi.

The normative design lives in RFC PR #19309. This tracker reflects the consistency bundle settled during review: mandatory RLI arbitration, continuation, source-instant markers, a verified contiguous frontier, replay-only rebuild catch-up, generation activation preconditions, and epoch-free one-commit LIRE publication.

Related: #18676

## Ground truth as of 2026-08-03

Static bootstrap and query paths have been validated at 10M and 1B scale, but mutation and maintenance are not complete:

- `VectorIndexer.buildUpdate` and `buildClean` are no-ops.
- Bootstrap writes generation `1` directly as `ACTIVE`.
- Persisted schemas and centroid keys still contain `centroidEpoch`.
- No source-instant marker family or manifest frontier exists.
- Tier-1 vector delta compaction and LIRE are not implemented.

Successful static benchmarks do not certify upsert correctness, freshness, rebuild under concurrent writers, compaction, or LIRE.

## Foundation PRs

- [ ] #19095 / PR #19309 — RFC document
- [ ] #19096 / PR #19310 — vector options and Spark DDL scaffolding
- [ ] #19097 / PR #19317 — MDT schema, keys, and payload model
- [ ] #19098 / PR #19318 — RaBitQ encoder/scorer contract
- [ ] #19099 — Spark bootstrap and CREATE INDEX execution

## Critical-path implementation order

### 1. Read-side correctness

- [ ] #19102 — approximate candidate generation
- Batched RLI finalist arbitration.
- Mandatory inverted-index validity gate.
- Continuation/refill retaining up to `maxRerankCandidates`.
- Overlay heap slack.
- Tombstone suppression and delta precedence.
- [ ] #19103 — exact rerank
- Batched RLI arbitration in exact mode.
- Continuation through candidate/rerank budgets.
- Authoritative base-table positional reads and explicit budget behavior.

### 2. Incremental maintenance

- [ ] #19498 — incremental `buildUpdate`, posting deltas, additive statistics, and atomic source markers

### 3. Freshness and replay

- [ ] #19499 — source-instant markers, verified contiguous frontier, planner policy, archival handling, and catch-up replay

### 4. Generation lifecycle

- [ ] #19100 — locked generation allocation, `BUILDING -> ACTIVE`, frontier-through-`T_cut` activation, and reader visibility
- [ ] #19104 — drop, abandoned/retired generation cleanup, and lifecycle recovery

### 5. Bounded growth

- [ ] #19501 — Tier-1 vector delta compaction, RLI dead-entry removal, locator refresh, checkpoint advancement, and marker pruning

### 6. Deferred geometry maintenance

- [ ] #19500 — epoch-free LIRE split/merge maintenance, one-MDT-commit publication, routing metrics, and skew-fixture certification

LIRE implementation is intentionally deferred behind Tier-1 compaction. Epoch removal is not deferred: it must land in #19097 / PR #19317 before the persisted format is established.

## Remaining query and integration work

- [ ] #19101 — centroid probing, posting-range derivation, prefix routing, and planner freshness integration
- [ ] #19105 — cross-feature certification and Spark SQL coverage

## Certifying scenarios

Feature issues own their unit and integration tests. #19105 additionally certifies the complete chain:

- Update in place.
- Delete absent from approximate and exact-rerank results.
- Vector-changing update moving clusters with old-posting suppression.
- Marker gap detection.
- No-op commit marker coverage.
- Archived-timeline frontier verification.
- Rebuild/catch-up under concurrent writers.
- Later, adversarial skew forcing LIRE split and merge.

## Tracking rules

- The RFC is the normative design; this issue and its children are the executable plan.
- Do not infer completion from code existing only on the integration branch.
- Close a child only after its focused upstream PR merges with its feature-level tests.
- Keep the internal Wibey shipping TODO as a mirror, not a second source of truth.
- Treat the historical RFC-104 TODO as superseded.

Start by reading RFC PR #19309 and the critical-path child issues, especially #19498, alongside the current VectorIndexer.buildUpdate and buildClean entry points. Use the feature issues' focused tests and #19105's Spark SQL certification scenarios to track progress. Done means the checklist's maintenance, freshness, lifecycle, compaction, LIRE, and certification work is implemented and its focused upstream PRs are merged.

Written by the indexing model from the issue text.