apache / apache/hudi

Add vector index generation visibility and reader consistency semantics

Open
#19,100 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Parent: #19094
Related: #18676

## Summary
Implement vector-index generation allocation, activation, and reader visibility under the freshness contract settled by RFC-109 review.

## Scope
- Allocate `generation = max(existing manifest generation) + 1` while holding the metadata lock.
- Write replacement generations as `BUILDING`, never directly as `ACTIVE`.
- Use replay-only catch-up shared with #19499; do not introduce a second rebuild mutation path.
- Define rebuild cut instant `T_cut` and require a verified contiguous frontier through `T_cut` before activation.
- Activate through the normal transaction path so generation state and visibility cannot race concurrent writers.
- Preserve the prior ACTIVE generation until replacement activation succeeds.
- Make readers select only a complete ACTIVE generation visible to their MDT snapshot.
- Define retry/idempotence behavior for interrupted build, catch-up, and activation.

## Acceptance criteria
- Bootstrap/rebuild no longer hardcodes generation `1` or writes `ACTIVE` directly.
- Incomplete or failed BUILDING generations are never reader-visible.
- A replacement cannot activate with a frontier gap through `T_cut`.
- Concurrent-writer rebuild tests prove no committed source instant is omitted.
- Tests cover max+1 allocation, competing/incomplete generations, failed activation, retry, and snapshot visibility.

## Dependencies
- #19498 incremental update hook.
- #19499 marker/frontier and catch-up replay.
- Schema/payload support in #19097 / PR #19317.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with RFC-109 review and parent #19094, then read dependency issues #19498 and #19499 and the schema/payload work in #19097 and PR #19317. Use the listed acceptance criteria to define done: generation allocation and activation must be retry-safe, incomplete generations must stay invisible, and concurrent-writer and snapshot-visibility tests must pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.