apache / apache/hudi

Add Flink support for HoodieSchema Variant data type

Open
#18,513 0 comments 0 reactions 1 assignee Claimed by @kbuci View on GitHub
type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Feature Description

(Will split into smaller tickets as we get agreement)
- Currently seems Flink ROW/RowType <-> Avro RECORD type handling already cover reading/writing Unshredded Variant fields from Flink in some code paths, by mapping it to a Flink ROW. We can instead update this handling to explicitly map HoodieSchema Variant to Flink 2.1's Variant type. The drawback though is that this will cause pre Flink 2.1 readers/writers to fail when handling Variant fields. But this can ensure that Flink DDL can create variant fields that correctly map to (HoodieSchema) Variant.
- Expand Flink variant read/writes to also handle shredded Variant types
- xpand Flink variant read/writes to also handle nesting Variant types in other types (like an array of variants)
- Integ tests for Flink SQL (with queries that use an array of floats). We should make sure we test with multiple versions of Flink 2.x and for both COW and MOR tables.
- Add functional/integration tests to ensure Spark can read variant data written by Flink writes.

### User Experience

**How users will use this feature:**
- Configuration changes needed
- API changes
- Usage examples

### Hudi RFC Requirements

**RFC PR link:** (if applicable)

**Why RFC is/isn't needed:**
- Does this change public interfaces/APIs? (Yes/No)
- Does this change storage format? (Yes/No)
- Justification:

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.