apache / apache/hudi

Add exact rerank from authoritative base-table vectors

Open
#19,103 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Parent: #19094
Related: #18676

## Summary
Implement exact rerank from authoritative base-table vectors with mandatory batched liveness arbitration and continuation.

## Scope
- Share MDT candidate generation and retain up to `maxRerankCandidates`.
- Batch candidate keys through RLI before authoritative vector fetch.
- Continue candidate scans when arbitration or positional reads invalidate finalists.
- Coalesce positional reads by file group and use stored row positions when valid.
- Handle MoR `rowPosition = -1` through the authoritative merged-read path.
- Score exact vectors and return final top-K.
- On stale index state or exhausted continuation/read budget, apply explicit `FAIL`, `WARN`, or exact-scan fallback policy; never silently present fewer than K as complete.

## Acceptance criteria
- Exact mode returns only authoritative live records.
- Deletes and cluster-changing updates cannot leak old postings into final results.
- Continuation retains the configured rerank candidate budget, not just K.
- Tests cover batched RLI arbitration, invalid locators, MoR log rows, duplicate/stale candidates, continuation, coalescing, and budget exhaustion.

## Dependencies
- #19102 candidate generation and shared continuation behavior.
- #19499 planner freshness contract for stale-policy decisions.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with dependency #19102 for candidate generation and shared continuation behavior, then read #19499 for the planner freshness contract and stale-policy decisions. Trace how batched RLI arbitration, authoritative reads, continuation, and reranking fit together; done means exact mode returns only live records and the listed tests cover invalidation, continuation, coalescing, and budget exhaustion.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.