Varaint Type: Add logical type annotation for writer
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 567
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 93
Description
### Is your feature request related to a problem or challenge?
Follow up to https://github.com/apache/iceberg-rust/pull/2188#discussion_r
iceberg-rust writes via `AsyncArrowWriter`, which derives the Parquet schema from the Arrow schema. In parquet 58.1.0, that path only emits the VARIANT annotation when the field carries the `parquet_variant_compute::VariantType` extension type and variant_experimental is enabled (otherwise logical_type_for_struct is a stub returning None). I couldn't find a public per-field hook to inject the annotation onto a plain `Struct(Binary,Binary)`.
So the real cost is: enable `variant_experimental` + attach the extension type to the field. Two risks that I see:
1. Turning on the feature may change how the reader decodes a VARIANT-annotated group (native VariantArray instead of Struct{metadata,value}) — could break the current read path that expects the struct.
2. New experimental dep surface.
Surfaced here: https://github.com/apache/iceberg-rust/pull/2188#discussion_r3326706646
### Describe the solution you'd like
_No response_
### Willingness to contribute
None
Contributor guide
Research direction
Start with the AsyncArrowWriter path and parquet 58.1.0's logical_type_for_struct behavior, then review the VariantType extension and the discussion in PR #2188. Determine how a per-field VARIANT annotation could be added while preserving the current Struct{metadata,value} read path; done means the risks around variant_experimental and the experimental dependency are resolved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100