apache / apache/hudi

Add drop, rebuild, and cleanup semantics for vector indexes

Open
#19,104 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Parent: #19094
Related: #18676

## Summary
Implement drop and cleanup semantics for MDT-backed vector indexes after generation activation behavior is established.

## Scope
- Vector-aware `DROP INDEX`.
- Cleanup of abandoned or failed BUILDING generations.
- Retirement and cleanup of superseded generations.
- Retention rules for generations still required by readable snapshots.
- Recovery and retry after interrupted lifecycle operations.
- Coordination with metadata cleaning and marker/checkpoint retention.

Rebuild encoding/catch-up belongs to #19499 and activation belongs to #19100; this issue links those flows rather than duplicating them.

## Acceptance criteria
- Drop removes index visibility safely and schedules physical cleanup.
- Failed/incomplete generations remain invisible and are eventually reclaimable.
- Superseded generations are not deleted while a retained snapshot may still reference them.
- Cleanup is idempotent and safe across retries.
- Tests cover drop, failed build cleanup, retired-generation retention, eventual cleanup, and interrupted lifecycle operations.

## Dependencies
- #19100 generation activation and visibility.
- #19499 freshness/frontier and replay.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading dependency issues #19100 and #19499 to understand generation visibility, freshness, and replay before tracing the existing vector-index lifecycle entry points. The payload names no files or tests; completion requires safe drop visibility, reclaimable failed and superseded generations, snapshot-aware retention, idempotent retries, and coverage for the listed lifecycle cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.