Add Flink support for HoodieSchema Variant data type
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Feature Description
(Will split into smaller tickets as we get agreement)
- Currently seems Flink ROW/RowType <-> Avro RECORD type handling already cover reading/writing Unshredded Variant fields from Flink in some code paths, by mapping it to a Flink ROW. We can instead update this handling to explicitly map HoodieSchema Variant to Flink 2.1's Variant type. The drawback though is that this will cause pre Flink 2.1 readers/writers to fail when handling Variant fields. But this can ensure that Flink DDL can create variant fields that correctly map to (HoodieSchema) Variant.
- Expand Flink variant read/writes to also handle shredded Variant types
- xpand Flink variant read/writes to also handle nesting Variant types in other types (like an array of variants)
- Integ tests for Flink SQL (with queries that use an array of floats). We should make sure we test with multiple versions of Flink 2.x and for both COW and MOR tables.
- Add functional/integration tests to ensure Spark can read variant data written by Flink writes.
### User Experience
**How users will use this feature:**
- Configuration changes needed
- API changes
- Usage examples
### Hudi RFC Requirements
**RFC PR link:** (if applicable)
**Why RFC is/isn't needed:**
- Does this change public interfaces/APIs? (Yes/No)
- Does this change storage format? (Yes/No)
- Justification:
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.