apache / apache/incubator-xtable
Add Variant Logical Type for Parquet Conversions
- Dominant language
- Java
- Stars
- 1.2k
- Forks
- 212
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 16
Description
### Feature Request / Improvement
A Variant is a binary encoding designed to store nested, schema-less data (JSON-like) while allowing for shredding—the ability to pull common fields out into their own columns for lightning-fast access.
Parquet users had to choose between two extremes:
1. Map/Struct types: High performance, but requires a fixed schema.
2. JSON Strings: Flexible, but requires scanning the whole string to find one value (very slow).
The Variant type is the modern "middle ground" solution, recently standardized by the Apache Parquet and Apache Iceberg communities.
This type is not yet supported for Syncing Parquet files into either Iceberg, Delta or Hudi.
Original PR supporting the current type into Parquet Format is:
[Added Variant to Parquet Format](https://github.com/apache/parquet-format/issues/459)
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue names no files, tests, or entry points. Start by reading the linked Parquet Format issue and then trace how Parquet syncing targets Iceberg, Delta, and Hudi; done means Variant values are supported when syncing into each of those formats.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100