truncate transform rejects binary values in transform_literal and LargeBinary arrays
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 567
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 93
Description
### Apache Iceberg Rust version
main (0.10.0)
### Describe the bug
The spec lists `binary` as a valid source type for `truncate[W]` ("Partition Transforms": `truncate[W]` source types are `int`, `long`, `decimal`, `string`, `binary`; "Truncate Transform Details": `binary` / `L`, length / `v.subarray(0, L)`). `Transform::result_type` agrees and accepts `PrimitiveType::Binary`, so a `truncate[W]` partition field on a binary column is a legal spec.
But both halves of `TransformFunction for Truncate` reject it:
1. `transform_literal` has no `PrimitiveLiteral::Binary` arm, so every predicate projection on such a field returns `FeatureUnsupported`. `InclusiveProjection` (used in `scan/cache.rs`) propagates that error, so any scan with a filter on that column fails instead of pruning.
2. `transform` handles `DataType::Binary` but not `DataType::LargeBinary`, and `schema_to_arrow_schema` maps Iceberg `binary` to `LargeBinary` (`crates/iceberg/src/arrow/schema.rs`, `PrimitiveType::Binary => DataType::LargeBinary`). So the array path fails on exactly the arrow type this crate produces for a binary column.
`Truncate::truncate_binary` already exists and is correct; it is just not wired to either entry point. pyiceberg and iceberg-go both implement truncate on binary.
### To Reproduce
```rust
let t = Transform::Truncate(3);
// 1. literal / projection path
let pred = /* binary column `b`, `b <= x'0102030405'` */;
t.project("pb", &pred);
// Err(FeatureUnsupported => Unsupported data type for truncate transform: Binary)
t.strict_project("pb", &pred);
// Err(FeatureUnsupported => Unsupported data type for truncate transform: Binary)
// 2. array path
let f = create_transform_function(&t).unwrap();
f.transform(Arc::new(BinaryArray::from_iter_values([vec![1u8,2,3,4,5]])));
// Ok([1, 2, 3])
f.transform(Arc::new(LargeBinaryArray::from_iter_values([vec![1u8,2,3,4,5]])));
// Err(FeatureUnsupported => Unsupported data type for truncate transform: LargeBinary)
```
### Expected behavior
`truncate[W]` on a binary column truncates to the first `W` bytes, in both the array and the literal path, so partition values can be computed and predicates can be projected.
### Willingness to contribute
I can contribute a fix for this bug independently
Contributor guide
Research direction
Start in the Truncate TransformFunction implementation, reading transform_literal and transform alongside truncate_binary, then check crates/iceberg/src/arrow/schema.rs to confirm Binary becomes LargeBinary. Exercise both the InclusiveProjection path used in scan/cache.rs and the array path with binary and LargeBinary inputs; done means truncate[W] handles both paths without FeatureUnsupported and returns the first W bytes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100