apache / apache/hudi

Add unit, integration, and Spark SQL coverage for vector indexes

Open
#19,105 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Parent: #19094
Related: #18676

## Summary
Provide cross-feature certification and user-facing Spark SQL coverage for RFC-109.

Feature-level tests remain in each implementation issue. This issue certifies the integrated system rather than becoming a late dumping ground for tests.

## Scope
- CREATE INDEX, search, rebuild, drop, and cleanup E2E flow.
- Approximate and exact-rerank behavior.
- Multibit compatibility.
- Upsert and freshness correctness across source and MDT timelines.
- Generation replacement under concurrent writers.
- Spark SQL examples matching the accepted API.
- Deferred LIRE skew certification after #19500 lands.

## Required certification scenarios
- Update in place is visible at the next eligible snapshot.
- Delete is absent in approximate and exact-rerank modes.
- Vector-changing update that moves clusters suppresses the old posting.
- Marker gap blocks frontier advancement and invokes configured stale policy.
- No-op commit receives marker coverage.
- Archived source instants are handled without an unverified frontier jump.
- Rebuild under concurrent writers activates only after verified catch-up through `T_cut`.
- Failed BUILDING generation remains invisible and recoverable.
- Adversarial skew fixtures deterministically force LIRE split and merge when that deferred service lands.

## Acceptance criteria
- Tests run against both COW and MoR where semantics differ.
- Failures are explicit; budget exhaustion or stale state never silently returns an incomplete result as complete.
- Examples and configuration names match the merged RFC/API.
- Static benchmark evidence is reported separately and is not treated as mutation certification.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading parent issue #19094, related issue #18676, and RFC-109 to understand the accepted API and integration boundaries. Map the CREATE INDEX, search, rebuild, drop, cleanup, Spark SQL, COW, and MoR certification scenarios before choosing test locations. Done means explicit coverage of the listed behaviors, including concurrent rebuilds, freshness, failed generations, and deferred LIRE skew certification after #19500.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, databases, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.