apache / apache/arrow-rs

Iceberg specific avro decoding

Open
#10,278 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

### Is your feature request related to a problem or challenge?

Iceberg has specific avro conventions: https://iceberg.apache.org/spec/#avro

One of these conventions state that the iceberg map type is translated to avro as an array of key-value records which they annotate with the logical `map` type.

### Describe the solution you'd like

One of the solutions is to propagate logical type annotations from avro to arrow metadata.
Reading such a custom avro file requires casting the resulting `Arrow List` type to `Arrow Map` type when the logical custom iceberg`map` type annotation is encountered, and this is only possible if we pass this annotation to the resulting arrow field.

Another solution is to handle this in during avro decoding, adding an `iceberg` feature flag.

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reading Iceberg's Avro conventions and the Avro decoding path; trace how an annotated map becomes an Arrow List and how field metadata is produced. Resolve whether logical-type propagation or an Iceberg feature flag is intended, then verify that Iceberg map annotations produce the expected Arrow Map behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.