apache / apache/hudi

Add Tier-1 vector delta compaction and locator refresh

Open
#19,501 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Parent: #19094
Related: #18676

## Summary
Implement Tier-1 maintenance for the vector metadata partition: fold accumulated deltas into packed posting blocks, repair locators, and checkpoint freshness metadata.

This pass bounds read amplification without changing centroid geometry. It is deliberately separate from LIRE split/merge maintenance.

## Scope
- Trigger maintenance from additive `deltaCount` and `tombstoneCount` thresholds.
- Merge posting deltas with surviving block contents per `(generation, cluster, shard)`.
- Join against the authoritative record-level index to remove dead or superseded postings.
- Repack survivors while preserving `(fileGroupId, rowPosition)` ordering.
- Reset delta/tombstone counters and update live statistics.
- Refresh locators for rewritten base-table file groups.
- Resolve MoR `rowPosition = -1` entries where physical positions become available.
- Advance a verified frontier checkpoint and prune markers only when checkpoint evidence makes pruning safe.
- Optionally produce an invalidation bitmap from this same evidence-gated pass; it is an optimization, not a correctness dependency.

## Correctness requirements
- Compaction cannot resurrect tombstoned or superseded postings.
- Repacked blocks preserve query and exact-rerank semantics.
- Marker pruning cannot destroy the only proof needed to verify frontier continuity.
- No centroid moves or cluster split/merge occurs in Tier 1.

## Acceptance criteria
- Read amplification remains bounded after sustained mutation.
- Tests cover mixed blocks/deltas/tombstones, cluster-changing updates, locator rewrites, MoR locator refinement, counter reset, checkpoint advancement, and safe marker pruning.
- Repacking is reusable as the write primitive for later LIRE maintenance.

## Dependencies
- Incremental vector-index update hook and additive statistics.
- Freshness marker/frontier contract.
- RLI arbitration and authoritative liveness checks.

## Blocks
- LIRE split/merge maintenance.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading parent issue #19094 and related issue #18676, then review the incremental vector-index update hook, freshness marker/frontier contract, and RLI arbitration requirements named here. The work is complete when Tier-1 compaction, locator refresh, counter reset, checkpoint advancement, and safe marker pruning satisfy the listed correctness and acceptance criteria.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.