feat: add variant type support (Iceberg v3)
- Dominant language
- Go
- Stars
- 463
- Forks
- 232
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 121
Description
### Feature Request / Improvement
Parent: #589 (v3 support tracking)
Iceberg v3 adds `variant` as a semi-structured type. Arrow-Go v18.5.2 (already in `go.mod`) has full encoding support via `parquet/variant` and `arrow/extensions.VariantType`.
This would thread variant through the type system, schema visitors, Arrow conversion, and expression binding. Just the non-shredded path (`struct`) for now. The goal would be to get the support first and follow up with shredding later.
I can work on this since I'm already doing the work on the Java API for shredding and other variant improvements.
### Related
- [Iceberg spec: Primitive Types](https://iceberg.apache.org/spec/#primitive-types)
- [Parquet Variant spec](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md)
- Arrow-Go: [`parquet/variant`](https://pkg.go.dev/github.com/apache/arrow-go/v18/parquet/variant), [`extensions#VariantType`](https://pkg.go.dev/github.com/apache/arrow-go/v18@v18.5.2/arrow/extensions#VariantType)
Contributor guide
Research direction
Start with the v3 support tracking issue #589 and inspect the existing type system, schema visitors, Arrow conversion, and expression binding paths. Compare the implementation points with Arrow-Go v18.5.2's parquet/variant and extensions.VariantType support. Done means variant works through those paths using only the non-shredded metadata-and-value representation, with shredding left for follow-up.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100