RFC-102 - Native Vector Search in Hudi
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Feature Description
**What the feature achieves:**
This feature provides the ability to perform a native **vector similarity search** on Hudi tables.
**Why this feature is needed:**
Building on [RFC-100](https://github.com/apache/hudi/pull/13924/files?short_path=a945f8d#diff-f05ae69c4f41edc32aabfbfc016a12ad1af72917314844f8ae52671234508c56) (unstructured data storage in Hudi), Hudi tables would contain unstructured content (e.g., images, video, documents) as well as the related *embeddings* for those contents. The next natural requirement for AI/ML workloads on Hudi is to **search these embeddings efficiently**:
### User Experience
**How users will use this feature:**
The initial scope of this feature was to be able to allow spark users the ability to perform vector search by providing a new `vector_search` SQL similar to other table value functions we have in Hudi. See the proposed RFC for more details: https://github.com/apache/hudi/pull/14218/changes.
### Hudi RFC Requirements
**RFC PR link:** See the proposed RFC for more details: https://github.com/apache/hudi/pull/14218/changes
**Why RFC is/isn't needed:**
- Does this change public interfaces/APIs? (Yes/No)
- Does this change storage format? (Yes/No)
- Justification:
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.