Add Tier-1 vector delta compaction and locator refresh
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Parent: #19094
Related: #18676
## Summary
Implement Tier-1 maintenance for the vector metadata partition: fold accumulated deltas into packed posting blocks, repair locators, and checkpoint freshness metadata.
This pass bounds read amplification without changing centroid geometry. It is deliberately separate from LIRE split/merge maintenance.
## Scope
- Trigger maintenance from additive `deltaCount` and `tombstoneCount` thresholds.
- Merge posting deltas with surviving block contents per `(generation, cluster, shard)`.
- Join against the authoritative record-level index to remove dead or superseded postings.
- Repack survivors while preserving `(fileGroupId, rowPosition)` ordering.
- Reset delta/tombstone counters and update live statistics.
- Refresh locators for rewritten base-table file groups.
- Resolve MoR `rowPosition = -1` entries where physical positions become available.
- Advance a verified frontier checkpoint and prune markers only when checkpoint evidence makes pruning safe.
- Optionally produce an invalidation bitmap from this same evidence-gated pass; it is an optimization, not a correctness dependency.
## Correctness requirements
- Compaction cannot resurrect tombstoned or superseded postings.
- Repacked blocks preserve query and exact-rerank semantics.
- Marker pruning cannot destroy the only proof needed to verify frontier continuity.
- No centroid moves or cluster split/merge occurs in Tier 1.
## Acceptance criteria
- Read amplification remains bounded after sustained mutation.
- Tests cover mixed blocks/deltas/tombstones, cluster-changing updates, locator rewrites, MoR locator refinement, counter reset, checkpoint advancement, and safe marker pruning.
- Repacking is reusable as the write primitive for later LIRE maintenance.
## Dependencies
- Incremental vector-index update hook and additive statistics.
- Freshness marker/frontier contract.
- RLI arbitration and authoritative liveness checks.
## Blocks
- LIRE split/merge maintenance.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading parent issue #19094 and related issue #18676, then review the incremental vector-index update hook, freshness marker/frontier contract, and RLI arbitration requirements named here. The work is complete when Tier-1 compaction, locator refresh, counter reset, checkpoint advancement, and safe marker pruning satisfy the listed correctness and acceptance criteria.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100