apache / apache/hudi

Add Flink support for HoodieSchema logical types (Vector, Blob, Variant)

Open
#18,505 0 comments 0 reactions 0 assignees View on GitHub
type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Feature Description

**What the feature achieves:**
Support enabling Flink engine readers and writers to be able to use the new HoodieSchema avro logical data types
- Vector types: https://github.com/apache/hudi/issues/18079
- Blob types: https://github.com/apache/hudi/issues/16856
- Variant types: https://github.com/apache/hudi/issues/17744

**Why this feature is needed:**
We want to be able to enable users to be able to read/write these new data types, for use cases like
- writing semistructured JSON data more efficiently (Variant)
- Updating/reading vector embeddings and performing vector search (Vector)
- Storing media data (Blob)
without needing to only use Spark

### Subtickets
- Variant https://github.com/apache/hudi/issues/18513
- Vector https://github.com/apache/hudi/issues/18506
- Blob https://github.com/apache/hudi/issues/18507

### User Experience

**How users will use this feature:**
- Configuration changes needed
- API changes
- Usage examples

### Hudi RFC Requirements

**RFC PR link:** (if applicable)

**Why RFC is/isn't needed:**
- Does this change public interfaces/APIs? (Yes/No)
- Does this change storage format? (Yes/No)
- Justification:

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with subtickets 18513, 18506, and 18507 to understand the separate Variant, Vector, and Blob requirements. Trace the Flink reader and writer entry points and determine the configuration, API, and usage-example expectations; done means Flink can read and write all three HoodieSchema logical types.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.