map Iceberg Binary to Arrow Binary instead of LargeBinary
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 567
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 93
Description
Follow up from https://github.com/apache/iceberg-rust/pull/2668
## Summary
iceberg-rust currently maps `PrimitiveType::Binary` to `DataType::LargeBinary` in the Arrow type conversion (`type_to_arrow_type`). This is inconsistent with:
1. **Java's implementation** — [ArrowSchemaUtil.java#L134](https://github.com/apache/iceberg/blob/main/arrow/src/main/java/org/apache/iceberg/arrow/ArrowSchemaUtil.java#L134) maps Iceberg Binary to Arrow Binary.
2. **parquet-rs behavior** — parquet-rs reads binary columns from Parquet files as `DataType::Binary`, not `LargeBinary`.
This mismatch causes unnecessary casts when reading Binary columns and creates inconsistency with the Java ecosystem.
## Scope
- Change `PrimitiveType::Binary` mapping from `DataType::LargeBinary` to `DataType::Binary` in `crates/iceberg/src/arrow/schema.rs`
- Update `create_primitive_array_repeated` in `crates/iceberg/src/arrow/value.rs` — the existing `DataType::Binary` arm already handles this, but the `LargeBinary` arm added in #2668 can be removed or kept as a fallback
- Audit all places that construct or expect `LargeBinary` arrays for Iceberg Binary columns
- Update tests that assert `LargeBinary` output
## Context
Raised in review of #2668 by @advancedxy. The `LargeBinary` arm in `create_primitive_array_repeated` was added to handle the current mapping for `_partition` columns partitioned by Binary. Once this mapping is corrected, that arm becomes unnecessary (but harmless to keep as a safety net).
This is a broader change that affects all Binary column reads, not just `_partition`.
Contributor guide
Research direction
Start in crates/iceberg/src/arrow/schema.rs at type_to_arrow_type, then inspect create_primitive_array_repeated in crates/iceberg/src/arrow/value.rs. Search the Arrow-related tests and codebase for LargeBinary expectations or constructors tied to Iceberg Binary columns, and run the affected tests. Done means Binary maps to Arrow Binary consistently and those tests pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 72/100