apache / apache/hudi

RFC-102 - Native Vector Search in Hudi

Open
#18,079 0 comments 0 reactions 1 assignee Claimed by @rahil-c View on GitHub
type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Feature Description

**What the feature achieves:**
This feature provides the ability to perform a native **vector similarity search** on Hudi tables.

**Why this feature is needed:**
Building on [RFC-100](https://github.com/apache/hudi/pull/13924/files?short_path=a945f8d#diff-f05ae69c4f41edc32aabfbfc016a12ad1af72917314844f8ae52671234508c56) (unstructured data storage in Hudi), Hudi tables would contain unstructured content (e.g., images, video, documents) as well as the related *embeddings* for those contents. The next natural requirement for AI/ML workloads on Hudi is to **search these embeddings efficiently**:

### User Experience

**How users will use this feature:**
The initial scope of this feature was to be able to allow spark users the ability to perform vector search by providing a new `vector_search` SQL similar to other table value functions we have in Hudi. See the proposed RFC for more details: https://github.com/apache/hudi/pull/14218/changes.

### Hudi RFC Requirements

**RFC PR link:** See the proposed RFC for more details: https://github.com/apache/hudi/pull/14218/changes

**Why RFC is/isn't needed:**
- Does this change public interfaces/APIs? (Yes/No)
- Does this change storage format? (Yes/No)
- Justification:

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.