Add vector-index freshness frontier and catch-up replay
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Parent: #19094
Related: #18676
## Summary
Implement the source-instant marker, verified contiguous frontier, planner policy, and catch-up replay contract for MDT-backed vector indexes.
The frontier is proof of contiguous source-timeline coverage, not merely the greatest instant observed. Missing coverage must never be mistaken for freshness.
## Scope
- Add source-instant marker records keyed by index identity and source instant.
- Track contiguous source coverage in the generation manifest using the concrete field name `lastContiguousSourceInstant`.
- Advance the frontier only after proving every required source instant is covered.
- Handle archived source instants when verifying continuity.
- Resolve `lastContiguousSourceInstant` with the pinned generation in `HoodieVectorSearchPlanBuilder`; for `FALLBACK`, reuse the existing `BruteForceSearchAlgorithm` full-table path rather than introducing a parallel fallback executor.
- Report marker-frontier lag as `indexLagInstants`.
- Implement catch-up replay using the same written-file reread and delta encoder as normal `buildUpdate`.
- Atomically write replayed deltas/statistics and their original source-instant markers.
- Detect and safely resume partial or interrupted replay.
## Correctness requirements
- A marker gap prevents frontier advancement beyond the gap.
- Maximum observed marker is never treated as contiguous coverage.
- No-op source commits are represented.
- Archived timeline segments do not permit unverified frontier jumps.
- Replay is idempotent and preserves source-instant identity.
## Acceptance criteria
- Planner tests cover fresh, stale, missing, and policy-specific behavior.
- Frontier tests cover gaps, no-op commits, archived instants, and interrupted replay.
- Catch-up shares the normal update implementation rather than introducing a second encoder.
- Metrics expose frontier lag and replay progress.
## Dependencies
- Schema/payload support in #19097 / PR #19317.
- Incremental vector-index update hook.
## Blocks
- Generation activation after rebuild.
- Marker pruning/checkpointing during Tier-1 compaction.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with parent #19094 and the schema/payload dependency in #19097/PR #19317, then trace HoodieVectorSearchPlanBuilder, BruteForceSearchAlgorithm, and the incremental vector-index update hook. Review the planner, frontier, and replay tests described in the acceptance criteria. Done means contiguous-frontier validation, idempotent catch-up replay, policy-specific planner behavior, and frontier-lag and replay-progress metrics are covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100