Add Flink support for HoodieSchema logical types (Vector, Blob, Variant)
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Feature Description
**What the feature achieves:**
Support enabling Flink engine readers and writers to be able to use the new HoodieSchema avro logical data types
- Vector types: https://github.com/apache/hudi/issues/18079
- Blob types: https://github.com/apache/hudi/issues/16856
- Variant types: https://github.com/apache/hudi/issues/17744
**Why this feature is needed:**
We want to be able to enable users to be able to read/write these new data types, for use cases like
- writing semistructured JSON data more efficiently (Variant)
- Updating/reading vector embeddings and performing vector search (Vector)
- Storing media data (Blob)
without needing to only use Spark
### Subtickets
- Variant https://github.com/apache/hudi/issues/18513
- Vector https://github.com/apache/hudi/issues/18506
- Blob https://github.com/apache/hudi/issues/18507
### User Experience
**How users will use this feature:**
- Configuration changes needed
- API changes
- Usage examples
### Hudi RFC Requirements
**RFC PR link:** (if applicable)
**Why RFC is/isn't needed:**
- Does this change public interfaces/APIs? (Yes/No)
- Does this change storage format? (Yes/No)
- Justification:
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with subtickets 18513, 18506, and 18507 to understand the separate Variant, Vector, and Blob requirements. Trace the Flink reader and writer entry points and determine the configuration, API, and usage-example expectations; done means Flink can read and write all three HoodieSchema logical types.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100