Add vector index generation visibility and reader consistency semantics
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Parent: #19094
Related: #18676
## Summary
Implement vector-index generation allocation, activation, and reader visibility under the freshness contract settled by RFC-109 review.
## Scope
- Allocate `generation = max(existing manifest generation) + 1` while holding the metadata lock.
- Write replacement generations as `BUILDING`, never directly as `ACTIVE`.
- Use replay-only catch-up shared with #19499; do not introduce a second rebuild mutation path.
- Define rebuild cut instant `T_cut` and require a verified contiguous frontier through `T_cut` before activation.
- Activate through the normal transaction path so generation state and visibility cannot race concurrent writers.
- Preserve the prior ACTIVE generation until replacement activation succeeds.
- Make readers select only a complete ACTIVE generation visible to their MDT snapshot.
- Define retry/idempotence behavior for interrupted build, catch-up, and activation.
## Acceptance criteria
- Bootstrap/rebuild no longer hardcodes generation `1` or writes `ACTIVE` directly.
- Incomplete or failed BUILDING generations are never reader-visible.
- A replacement cannot activate with a frontier gap through `T_cut`.
- Concurrent-writer rebuild tests prove no committed source instant is omitted.
- Tests cover max+1 allocation, competing/incomplete generations, failed activation, retry, and snapshot visibility.
## Dependencies
- #19498 incremental update hook.
- #19499 marker/frontier and catch-up replay.
- Schema/payload support in #19097 / PR #19317.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with RFC-109 review and parent #19094, then read dependency issues #19498 and #19499 and the schema/payload work in #19097 and PR #19317. Use the listed acceptance criteria to define done: generation allocation and activation must be retry-safe, incomplete generations must stay invisible, and concurrent-writer and snapshot-visibility tests must pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100