apache / apache/hudi

RFC-102 - Native Vector Search in Hudi

Open
#14,219 0 comments 0 reactions 0 assignees View on GitHub
type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Feature Description

#### What the feature achieves:

This feature enables native vector similarity search capabilities directly on Hudi tables. It allows users to store, manage, and query vector embeddings (e.g., from text, image, or audio models) alongside structured data, and perform nearest-neighbor searches using distance metrics such as cosine, dot product, or Euclidean distance — all within Hudi tables. This brings AI/ML-centric search workloads (semantic, multimodal, or embedding-based retrieval) natively into the Hudi lakehouse.

#### Why this feature is needed:

Modern data lakes increasingly store unstructured or multimodal data (text, images, video) with associated embeddings for retrieval and ranking. Today, vector search is typically performed outside the lakehouse using specialized vector databases, leading to data duplication, inconsistency, and complex pipelines. Adding native vector search to Hudi unifies structured and vector data management, reduces latency between ingestion and retrieval, and enables scalable AI/ML workflows directly on the lakehouse without external dependencies.

### User Experience

**How users will use this feature:**
Please read RFC 102: https://github.com/apache/hudi/pull/14218

### Hudi RFC Requirements

**RFC PR link:** (if applicable)
https://github.com/apache/hudi/pull/14218

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading RFC 102 in pull request 14218, which defines the proposed native vector search feature for Hudi tables. Use that RFC to determine the implementation scope and acceptance criteria; done should cover storing, managing, and querying embeddings with the stated similarity metrics within Hudi.

Written by the indexing model from the issue text.

Assessment

Tech stack
machine-learning
Domain
data-engineering, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.