RFC-104: Add Vector Index Implementation within Hudi
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Feature Description
**What the feature achieves:**
Big thanks to @chrevanthreddy who helped with initial research and design for an approximate nearest neighbor (ANN) index for embedding columns within Hudi tables.
The design leverages a lightweight cluster-routing index in the Hudi metadata table provides file-group pruning, as well as storing RaBitQ binary codes and scalars for fast scanning. It builds on research from the following SPFresh paper: https://arxiv.org/abs/2410.14452
This github "epic" issue tracks all the work regarding this intiative.
See the discussion below: https://github.com/apache/hudi/discussions/18500 for more details
**Why this feature is needed:**
Performance Optimization for user to speed up existing vector search
### User Experience
**How users will use this feature:**
- Configuration changes needed
- API changes
- Usage examples
### Hudi RFC Requirements
**RFC PR link:** (if applicable)
**Why RFC is/isn't needed:**
- Does this change public interfaces/APIs? (Yes/No)
- Does this change storage format? (Yes/No)
- Justification:
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the feature description and the linked discussion at github.com/apache/hudi/discussions/18500, then review the SPFresh paper cited in the issue. The issue describes an ANN vector index using Hudi metadata, file-group pruning, and RaBitQ codes, but does not identify files, tests, or a bounded deliverable; completion criteria need to be defined before implementation can begin.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100