apache / apache/iceberg-rust

truncate transform rejects binary values in transform_literal and LargeBinary arrays

Open
#3,119 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

### Apache Iceberg Rust version

main (0.10.0)

### Describe the bug

The spec lists `binary` as a valid source type for `truncate[W]` ("Partition Transforms": `truncate[W]` source types are `int`, `long`, `decimal`, `string`, `binary`; "Truncate Transform Details": `binary` / `L`, length / `v.subarray(0, L)`). `Transform::result_type` agrees and accepts `PrimitiveType::Binary`, so a `truncate[W]` partition field on a binary column is a legal spec.

But both halves of `TransformFunction for Truncate` reject it:

1. `transform_literal` has no `PrimitiveLiteral::Binary` arm, so every predicate projection on such a field returns `FeatureUnsupported`. `InclusiveProjection` (used in `scan/cache.rs`) propagates that error, so any scan with a filter on that column fails instead of pruning.
2. `transform` handles `DataType::Binary` but not `DataType::LargeBinary`, and `schema_to_arrow_schema` maps Iceberg `binary` to `LargeBinary` (`crates/iceberg/src/arrow/schema.rs`, `PrimitiveType::Binary => DataType::LargeBinary`). So the array path fails on exactly the arrow type this crate produces for a binary column.

`Truncate::truncate_binary` already exists and is correct; it is just not wired to either entry point. pyiceberg and iceberg-go both implement truncate on binary.

### To Reproduce

```rust
let t = Transform::Truncate(3);

// 1. literal / projection path
let pred = /* binary column `b`, `b <= x'0102030405'` */;
t.project("pb", &pred);
// Err(FeatureUnsupported => Unsupported data type for truncate transform: Binary)
t.strict_project("pb", &pred);
// Err(FeatureUnsupported => Unsupported data type for truncate transform: Binary)

// 2. array path
let f = create_transform_function(&t).unwrap();
f.transform(Arc::new(BinaryArray::from_iter_values([vec![1u8,2,3,4,5]])));
// Ok([1, 2, 3])
f.transform(Arc::new(LargeBinaryArray::from_iter_values([vec![1u8,2,3,4,5]])));
// Err(FeatureUnsupported => Unsupported data type for truncate transform: LargeBinary)
```

### Expected behavior

`truncate[W]` on a binary column truncates to the first `W` bytes, in both the array and the literal path, so partition values can be computed and predicates can be projected.

### Willingness to contribute

I can contribute a fix for this bug independently

Contributor guide

Open the contributing guide

Research direction

Start in the Truncate TransformFunction implementation, reading transform_literal and transform alongside truncate_binary, then check crates/iceberg/src/arrow/schema.rs to confirm Binary becomes LargeBinary. Exercise both the InclusiveProjection path used in scan/cache.rs and the array path with binary and LargeBinary inputs; done means truncate[W] handles both paths without FeatureUnsupported and returns the first W bytes.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.