apache / apache/iceberg-rust

map Iceberg Binary to Arrow Binary instead of LargeBinary

Open
#2,698 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

Follow up from https://github.com/apache/iceberg-rust/pull/2668

## Summary

iceberg-rust currently maps `PrimitiveType::Binary` to `DataType::LargeBinary` in the Arrow type conversion (`type_to_arrow_type`). This is inconsistent with:

1. **Java's implementation** — [ArrowSchemaUtil.java#L134](https://github.com/apache/iceberg/blob/main/arrow/src/main/java/org/apache/iceberg/arrow/ArrowSchemaUtil.java#L134) maps Iceberg Binary to Arrow Binary.
2. **parquet-rs behavior** — parquet-rs reads binary columns from Parquet files as `DataType::Binary`, not `LargeBinary`.

This mismatch causes unnecessary casts when reading Binary columns and creates inconsistency with the Java ecosystem.

## Scope

- Change `PrimitiveType::Binary` mapping from `DataType::LargeBinary` to `DataType::Binary` in `crates/iceberg/src/arrow/schema.rs`
- Update `create_primitive_array_repeated` in `crates/iceberg/src/arrow/value.rs` — the existing `DataType::Binary` arm already handles this, but the `LargeBinary` arm added in #2668 can be removed or kept as a fallback
- Audit all places that construct or expect `LargeBinary` arrays for Iceberg Binary columns
- Update tests that assert `LargeBinary` output

## Context

Raised in review of #2668 by @advancedxy. The `LargeBinary` arm in `create_primitive_array_repeated` was added to handle the current mapping for `_partition` columns partitioned by Binary. Once this mapping is corrected, that arm becomes unnecessary (but harmless to keep as a safety net).

This is a broader change that affects all Binary column reads, not just `_partition`.

Contributor guide

Open the contributing guide

Research direction

Start in crates/iceberg/src/arrow/schema.rs at type_to_arrow_type, then inspect create_primitive_array_repeated in crates/iceberg/src/arrow/value.rs. Search the Arrow-related tests and codebase for LargeBinary expectations or constructors tied to Iceberg Binary columns, and run the affected tests. Done means Binary maps to Arrow Binary consistently and those tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.