apache / apache/hudi

Add vector-index freshness frontier and catch-up replay

Open
#19,499 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Parent: #19094
Related: #18676

## Summary
Implement the source-instant marker, verified contiguous frontier, planner policy, and catch-up replay contract for MDT-backed vector indexes.

The frontier is proof of contiguous source-timeline coverage, not merely the greatest instant observed. Missing coverage must never be mistaken for freshness.

## Scope
- Add source-instant marker records keyed by index identity and source instant.
- Track contiguous source coverage in the generation manifest using the concrete field name `lastContiguousSourceInstant`.
- Advance the frontier only after proving every required source instant is covered.
- Handle archived source instants when verifying continuity.
- Resolve `lastContiguousSourceInstant` with the pinned generation in `HoodieVectorSearchPlanBuilder`; for `FALLBACK`, reuse the existing `BruteForceSearchAlgorithm` full-table path rather than introducing a parallel fallback executor.
- Report marker-frontier lag as `indexLagInstants`.
- Implement catch-up replay using the same written-file reread and delta encoder as normal `buildUpdate`.
- Atomically write replayed deltas/statistics and their original source-instant markers.
- Detect and safely resume partial or interrupted replay.

## Correctness requirements
- A marker gap prevents frontier advancement beyond the gap.
- Maximum observed marker is never treated as contiguous coverage.
- No-op source commits are represented.
- Archived timeline segments do not permit unverified frontier jumps.
- Replay is idempotent and preserves source-instant identity.

## Acceptance criteria
- Planner tests cover fresh, stale, missing, and policy-specific behavior.
- Frontier tests cover gaps, no-op commits, archived instants, and interrupted replay.
- Catch-up shares the normal update implementation rather than introducing a second encoder.
- Metrics expose frontier lag and replay progress.

## Dependencies
- Schema/payload support in #19097 / PR #19317.
- Incremental vector-index update hook.

## Blocks
- Generation activation after rebuild.
- Marker pruning/checkpointing during Tier-1 compaction.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with parent #19094 and the schema/payload dependency in #19097/PR #19317, then trace HoodieVectorSearchPlanBuilder, BruteForceSearchAlgorithm, and the incremental vector-index update hook. Review the planner, frontier, and replay tests described in the acceptance criteria. Done means contiguous-frontier validation, idempotent catch-up replay, policy-specific planner behavior, and frontier-lag and replay-progress metrics are covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.