Implement incremental maintenance for MDT-backed vector indexes
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Parent: #19094
Related: #18676
## Summary
Implement incremental maintenance for the MDT-backed vector index after each committed base-table write.
This is the writer-side critical path for keeping an active vector generation current. It follows the existing `EXPRESSION_INDEX` written-file reread pattern rather than relying on incomplete write-status payloads.
## Scope
- Ensure vector-index update dispatch reaches `VectorIndexer.buildUpdate` for every completed source instant, including commits with no changed vector rows.
- Re-read files written by the source commit and extract authoritative vector rows and locators.
- Load the ACTIVE generation's centroids and quantizer metadata once per update.
- Assign and encode changed vectors against that ACTIVE generation.
- Emit posting deltas and additive `ClusterStats` updates.
- Emit the source-instant marker in the same MDT update as its posting/stat changes.
- Represent MoR log-resident rows with `rowPosition = -1` until locator refresh.
- Preserve idempotence for replay of an already-marked source instant.
## Correctness requirements
- Update-in-place suppresses the old posting and exposes the replacement.
- A vector-changing update that moves clusters cannot leave the old cluster posting visible.
- Deletes do not remain visible through either approximate or exact-rerank search.
- Marker publication is atomic with the corresponding vector-index changes.
- A no-op source commit still produces marker coverage.
## Acceptance criteria
- `VectorIndexer.buildUpdate` is no longer a no-op.
- COW and MoR update paths are covered.
- Posting deltas, additive statistics, and source markers are committed together.
- Reprocessing is idempotent.
- Unit/integration tests cover insert, update-in-place, cluster-changing update, delete, no-op commit, and MoR log-row location semantics.
## Dependencies
- Schema/payload support in #19097 / PR #19317.
- Read-side arbitration and continuation work in #19102 and #19103.
## Blocks
- Freshness frontier and catch-up replay.
- Generation rebuild catch-up and activation.
- Tier-1 vector compaction.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at VectorIndexer.buildUpdate and trace the existing EXPRESSION_INDEX written-file reread pattern for each completed source instant. Compare the COW and MoR update paths, then inspect the schema/payload support from #19097 / PR #19317 and the read-side work in #19102 and #19103. Done means atomic posting, statistics, and marker updates with idempotent replay and coverage for the listed insert, update, delete, no-op, and MoR cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, distributed-systems, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100