apache / apache/iceberg-rust

Varaint Type: Add logical type annotation for writer

Open
#2,546 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

### Is your feature request related to a problem or challenge?

Follow up to https://github.com/apache/iceberg-rust/pull/2188#discussion_r

iceberg-rust writes via `AsyncArrowWriter`, which derives the Parquet schema from the Arrow schema. In parquet 58.1.0, that path only emits the VARIANT annotation when the field carries the `parquet_variant_compute::VariantType` extension type and variant_experimental is enabled (otherwise logical_type_for_struct is a stub returning None). I couldn't find a public per-field hook to inject the annotation onto a plain `Struct(Binary,Binary)`.

So the real cost is: enable `variant_experimental` + attach the extension type to the field. Two risks that I see:
1. Turning on the feature may change how the reader decodes a VARIANT-annotated group (native VariantArray instead of Struct{metadata,value}) — could break the current read path that expects the struct.
2. New experimental dep surface.

Surfaced here: https://github.com/apache/iceberg-rust/pull/2188#discussion_r3326706646

### Describe the solution you'd like

_No response_

### Willingness to contribute

None

Contributor guide

Open the contributing guide

Research direction

Start with the AsyncArrowWriter path and parquet 58.1.0's logical_type_for_struct behavior, then review the VariantType extension and the discussion in PR #2188. Determine how a per-field VARIANT annotation could be added while preserving the current Struct{metadata,value} read path; done means the risks around variant_experimental and the experimental dependency are resolved.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.