apache / apache/hudi

Add Flink support for Blob data type

Open
#18,507 6 comments 0 reactions 1 assignee Claimed by @kbuci View on GitHub
type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Feature Description

Feature Description
(Will split into smaller tickets as we get agreement)

All Flink read/write related classes should handle HoodieSchema Blob type.
- Based on initial scan, it seems Flink ROW/RowType <-> Avro RECORD type handling might already cover reading/writing Blob fields from Flink. Do we need to add more specific handling for exposing the HoodieSchema blob logical type when creating Flink ROW/RowType? Like passing in `"symbols": ["INLINE", "OUT_OF_LINE"]` ?
- Check if we need to add more Integ tests for Flink SQL We should make sure we test with multiple versions of Flink 2.x and for both COW and MOR tables.
- Add functional/integration tests to ensure Flink can read blob data records written by Spark writes, and vice versa.
- Exploration/comparison with Ray based on RFC https://github.com/apache/hudi/blob/master/rfc/rfc-100/rfc-100.md - should we even support (HUDI) Flink with blob if users can integrate with (HUDI) Ray for low-latency use cases?

- Enable read_blob for FlinkSQL for OOL blobs https://github.com/apache/hudi/issues/19032

### User Experience

**How users will use this feature:**
- Configuration changes needed
- API changes
- Usage examples

### Hudi RFC Requirements

**RFC PR link:** (if applicable)

**Why RFC is/isn't needed:**
- Does this change public interfaces/APIs? (Yes/No)
- Does this change storage format? (Yes/No)
- Justification:

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.