Iceberg specific avro decoding
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
### Is your feature request related to a problem or challenge?
Iceberg has specific avro conventions: https://iceberg.apache.org/spec/#avro
One of these conventions state that the iceberg map type is translated to avro as an array of key-value records which they annotate with the logical `map` type.
### Describe the solution you'd like
One of the solutions is to propagate logical type annotations from avro to arrow metadata.
Reading such a custom avro file requires casting the resulting `Arrow List` type to `Arrow Map` type when the logical custom iceberg`map` type annotation is encountered, and this is only possible if we pass this annotation to the resulting arrow field.
Another solution is to handle this in during avro decoding, adding an `iceberg` feature flag.
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start by reading Iceberg's Avro conventions and the Avro decoding path; trace how an annotated map becomes an Arrow List and how field metadata is produced. Resolve whether logical-type propagation or an Iceberg feature flag is intended, then verify that Iceberg map annotations produce the expected Arrow Map behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100