apache / apache/hudi

RFC-104: Add Vector Index Implementation within Hudi

Open
#18,676 1 comment 0 reactions 0 assignees View on GitHub
type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Feature Description

**What the feature achieves:**

Big thanks to @chrevanthreddy who helped with initial research and design for an approximate nearest neighbor (ANN) index for embedding columns within Hudi tables.

The design leverages a lightweight cluster-routing index in the Hudi metadata table provides file-group pruning, as well as storing RaBitQ binary codes and scalars for fast scanning. It builds on research from the following SPFresh paper: https://arxiv.org/abs/2410.14452

This github "epic" issue tracks all the work regarding this intiative.
See the discussion below: https://github.com/apache/hudi/discussions/18500 for more details

**Why this feature is needed:**
Performance Optimization for user to speed up existing vector search

### User Experience

**How users will use this feature:**
- Configuration changes needed
- API changes
- Usage examples

### Hudi RFC Requirements

**RFC PR link:** (if applicable)

**Why RFC is/isn't needed:**
- Does this change public interfaces/APIs? (Yes/No)
- Does this change storage format? (Yes/No)
- Justification:

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the feature description and the linked discussion at github.com/apache/hudi/discussions/18500, then review the SPFresh paper cited in the issue. The issue describes an ANN vector index using Hudi metadata, file-group pruning, and RaBitQ codes, but does not identify files, tests, or a bounded deliverable; completion criteria need to be defined before implementation can begin.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.