apache / apache/hudi

Add MDT posting scan and approximate top-K candidate generation

Open
#19,102 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Parent: #19094
Related: #18676

## Summary
Implement approximate candidate generation from MDT vector postings with mandatory liveness arbitration and bounded continuation.

## Scope
- Scan selected posting ranges and score RaBitQ candidates.
- Support the agreed multibit behavior.
- Batch candidate keys through the record-level index (RLI).
- Treat the inverted-index gate as mandatory: stale or unavailable validation follows configured policy rather than silently accepting candidates.
- Suppress tombstoned block/delta entries.
- Apply correct delta precedence over base posting blocks.
- Size candidate heaps with overlay slack so landing deltas cannot evict valid base candidates before merge.
- Continue scanning/refilling when arbitration removes finalists.
- Retain up to `maxRerankCandidates` across continuation batches; do not retain only K.
- Stop on sufficient live candidates, exhausted ranges, or explicit request budget.

## Acceptance criteria
- Approximate mode cannot return stale/deleted candidates merely because they scored highly.
- Continuation fills K live results when candidates remain within budget.
- Budget exhaustion is explicit and policy-controlled.
- Tests cover stale RLI state, tombstones, base/delta precedence, overlay pressure, duplicate keys, continuation retention, and fewer-than-K exhaustion.

## Dependencies
- Schema/payload support in #19097.
- Planner/range work in #19101.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading parent issue #19094 and dependency issues #19097 and #19101 to understand the schema, payload, planner, and range contracts. Then map the MDT posting scan, RLI arbitration, and continuation flow described here. Done means stale, tombstoned, duplicate, and precedence cases are handled, continuation fills live results within budget, and the listed acceptance tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.