apache / apache/iceberg-go

feat: add variant type support (Iceberg v3)

Open
#929 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
463
Forks
232
Avg merge
2d 11h
Merged PRs (30d)
121

Description

### Feature Request / Improvement

Parent: #589 (v3 support tracking)

Iceberg v3 adds `variant` as a semi-structured type. Arrow-Go v18.5.2 (already in `go.mod`) has full encoding support via `parquet/variant` and `arrow/extensions.VariantType`.

This would thread variant through the type system, schema visitors, Arrow conversion, and expression binding. Just the non-shredded path (`struct`) for now. The goal would be to get the support first and follow up with shredding later.

I can work on this since I'm already doing the work on the Java API for shredding and other variant improvements.

### Related

- [Iceberg spec: Primitive Types](https://iceberg.apache.org/spec/#primitive-types)
- [Parquet Variant spec](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md)
- Arrow-Go: [`parquet/variant`](https://pkg.go.dev/github.com/apache/arrow-go/v18/parquet/variant), [`extensions#VariantType`](https://pkg.go.dev/github.com/apache/arrow-go/v18@v18.5.2/arrow/extensions#VariantType)

Contributor guide

Open the contributing guide

Research direction

Start with the v3 support tracking issue #589 and inspect the existing type system, schema visitors, Arrow conversion, and expression binding paths. Compare the implementation points with Arrow-Go v18.5.2's parquet/variant and extensions.VariantType support. Done means variant works through those paths using only the non-shredded metadata-and-value representation, with shredding left for follow-up.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.