apache / apache/hudi

Implement incremental maintenance for MDT-backed vector indexes

Open
#19,498 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Parent: #19094
Related: #18676

## Summary
Implement incremental maintenance for the MDT-backed vector index after each committed base-table write.

This is the writer-side critical path for keeping an active vector generation current. It follows the existing `EXPRESSION_INDEX` written-file reread pattern rather than relying on incomplete write-status payloads.

## Scope
- Ensure vector-index update dispatch reaches `VectorIndexer.buildUpdate` for every completed source instant, including commits with no changed vector rows.
- Re-read files written by the source commit and extract authoritative vector rows and locators.
- Load the ACTIVE generation's centroids and quantizer metadata once per update.
- Assign and encode changed vectors against that ACTIVE generation.
- Emit posting deltas and additive `ClusterStats` updates.
- Emit the source-instant marker in the same MDT update as its posting/stat changes.
- Represent MoR log-resident rows with `rowPosition = -1` until locator refresh.
- Preserve idempotence for replay of an already-marked source instant.

## Correctness requirements
- Update-in-place suppresses the old posting and exposes the replacement.
- A vector-changing update that moves clusters cannot leave the old cluster posting visible.
- Deletes do not remain visible through either approximate or exact-rerank search.
- Marker publication is atomic with the corresponding vector-index changes.
- A no-op source commit still produces marker coverage.

## Acceptance criteria
- `VectorIndexer.buildUpdate` is no longer a no-op.
- COW and MoR update paths are covered.
- Posting deltas, additive statistics, and source markers are committed together.
- Reprocessing is idempotent.
- Unit/integration tests cover insert, update-in-place, cluster-changing update, delete, no-op commit, and MoR log-row location semantics.

## Dependencies
- Schema/payload support in #19097 / PR #19317.
- Read-side arbitration and continuation work in #19102 and #19103.

## Blocks
- Freshness frontier and catch-up replay.
- Generation rebuild catch-up and activation.
- Tier-1 vector compaction.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at VectorIndexer.buildUpdate and trace the existing EXPRESSION_INDEX written-file reread pattern for each completed source instant. Compare the COW and MoR update paths, then inspect the schema/payload support from #19097 / PR #19317 and the read-side work in #19102 and #19103. Done means atomic posting, statistics, and marker updates with idempotent replay and coverage for the listed insert, update, delete, no-op, and MoR cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.