Add MDT posting scan and approximate top-K candidate generation
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Parent: #19094
Related: #18676
## Summary
Implement approximate candidate generation from MDT vector postings with mandatory liveness arbitration and bounded continuation.
## Scope
- Scan selected posting ranges and score RaBitQ candidates.
- Support the agreed multibit behavior.
- Batch candidate keys through the record-level index (RLI).
- Treat the inverted-index gate as mandatory: stale or unavailable validation follows configured policy rather than silently accepting candidates.
- Suppress tombstoned block/delta entries.
- Apply correct delta precedence over base posting blocks.
- Size candidate heaps with overlay slack so landing deltas cannot evict valid base candidates before merge.
- Continue scanning/refilling when arbitration removes finalists.
- Retain up to `maxRerankCandidates` across continuation batches; do not retain only K.
- Stop on sufficient live candidates, exhausted ranges, or explicit request budget.
## Acceptance criteria
- Approximate mode cannot return stale/deleted candidates merely because they scored highly.
- Continuation fills K live results when candidates remain within budget.
- Budget exhaustion is explicit and policy-controlled.
- Tests cover stale RLI state, tombstones, base/delta precedence, overlay pressure, duplicate keys, continuation retention, and fewer-than-K exhaustion.
## Dependencies
- Schema/payload support in #19097.
- Planner/range work in #19101.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading parent issue #19094 and dependency issues #19097 and #19101 to understand the schema, payload, planner, and range contracts. Then map the MDT posting scan, RLI arbitration, and continuation flow described here. Done means stale, tombstoned, duplicate, and precedence cases are handled, continuation fills live results within budget, and the listed acceptance tests pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100