apache / apache/incubator-xtable

Add Variant Logical Type for Parquet Conversions

Open
#804 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
1.2k
Forks
212
Avg merge
4d 9h
Merged PRs (30d)
16

Description

### Feature Request / Improvement

A Variant is a binary encoding designed to store nested, schema-less data (JSON-like) while allowing for shredding—the ability to pull common fields out into their own columns for lightning-fast access.

Parquet users had to choose between two extremes:

1. Map/Struct types: High performance, but requires a fixed schema.
2. JSON Strings: Flexible, but requires scanning the whole string to find one value (very slow).

The Variant type is the modern "middle ground" solution, recently standardized by the Apache Parquet and Apache Iceberg communities.

This type is not yet supported for Syncing Parquet files into either Iceberg, Delta or Hudi.

Original PR supporting the current type into Parquet Format is:

[Added Variant to Parquet Format](https://github.com/apache/parquet-format/issues/459)

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names no files, tests, or entry points. Start by reading the linked Parquet Format issue and then trace how Parquet syncing targets Iceberg, Delta, and Hudi; done means Variant values are supported when syncing into each of those formats.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.