apache / apache/hudi

Add support for Variant

Open
#17,744 6 comments 0 reactions 1 assignee Claimed by @voonhous View on GitHub
type:epic type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Feature Description

**What the feature achieves:**
Hudi readers and writers should be able to handle datasets with variant types. This will allow users to work with semi-structured data more easily.

**Why this feature is needed:**
The variant type is now formally defined in Parquet and engines like Spark have full support for this type. Users with semi-structured data are otherwise forced to use strings or byte arrays to store this data.

### User Experience

**How users will use this feature:**
- Configuration changes needed
- API changes
- Usage examples

The user will be able to add a Variant field to the schema and write data with this type using an engine like Spark where the data type is already supported.

Readers should be able to return the variant type for engines that support it or otherwise return the underlying struct with the raw fields.

The schema should be parseable by older Hudi readers for compatibility.

### Hudi RFC Requirements

**RFC PR link:** (if applicable)
This is an extension of RFC-99 which introduces the type system for Hudi. This will be the first type added that is not already supported in Avro.

**Why RFC is/isn't needed:**
- Does this change public interfaces/APIs? (Yes/No)
- Does this change storage format? (Yes/No)
- Justification:

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.