apache / apache/iceberg-rust

Truncate transform diverges from Iceberg Java's TruncateUtil for widths above 2^30, and panics in debug builds at the integer minimum

Open
#3,141 0 comments 2 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

### Apache Iceberg Rust version

main @ `3d84c81353b1b23b6e4ae8eea8f8a021cc6927a7`

### Describe the bug

`Truncate::transform` in `crates/iceberg/src/transform/truncate.rs` does not reproduce Iceberg Java's `TruncateUtil` arithmetic. There are two distinct problems.

**1. `truncate_i32` diverges from Java for widths above 2^30**

```rust
fn truncate_i32(v: i32, width: i32) -> i32 {
v - v.rem_euclid(width)
}
```

Java computes the positive remainder with an expression that is allowed to overflow:

```java
// org.apache.iceberg.util.TruncateUtil
public static int truncateInt(int width, int value) {
return value - (((value % width) + width) % width);
}
```

`(value % width) + width` overflows `int` whenever `value % width > Integer.MAX_VALUE - width`, which is reachable for any `width > 2^30`. `rem_euclid` never overflows, so the two produce different partition values. Only positive values with a large width are affected; negative values and widths at or below 2^30 agree.

`Truncate.get(int width)` in Iceberg Java only validates `width > 0`, so these widths are accepted by the spec. Whether Java's overflow is desirable is beside the point — it is the value iceberg-java writes into the partition tuple, so a Rust writer that computes something else produces a table the Java reader partitions differently.

**2. Overflow panics in debug builds at the integer minimum**

```rust
fn truncate_i64(v: i64, width: i64) -> i64 {
v - (((v % width) + width) % width)
}
```

For `v = i64::MIN, width = 1000` the remainder is 192 and `v - 192` underflows. With `debug_assertions` on this panics (`attempt to subtract with overflow`); in a release build it wraps and matches Java exactly. `truncate_i32` has the same shape at `i32::MIN`, and `truncate_decimal_i128` uses the same unchecked subtraction (unreachable for a 38-digit Iceberg decimal, but the same pattern).

A panic here aborts the write task, so a debug or test build of any downstream writer fails on data that Java handles.

### To Reproduce

Values below were produced by running `TruncateUtil` from apache-iceberg 1.11.0 on JDK 17 against the Rust expressions:

| width | value | Java `TruncateUtil.truncateInt` | iceberg-rust `truncate_i32` |
| --- | --- | --- | --- |
| 2000000000 | 1500000000 | -2000000000 | 0 |
| 2147483647 | 2147483646 | -2147483647 | 0 |
| 1073741824 | 1500000000 | 1073741824 | 1073741824 |
| 1000 | 2147483647 | 2147483000 | 2147483000 |

For problem 2, in a debug build:

```rust
let t = create_transform_function(&Transform::Truncate(1000)).unwrap();
t.transform(Arc::new(Int64Array::from(vec![i64::MIN]))).unwrap(); // panics
```

### Expected behavior

Both kernels bit-identical to Java in debug and release. Wrapping arithmetic gives that:

```rust
fn truncate_i32(v: i32, w: i32) -> i32 {
v.wrapping_sub((v % w).wrapping_add(w) % w)
}

fn truncate_i64(v: i64, w: i64) -> i64 {
v.wrapping_sub((v % w).wrapping_add(w) % w)
}
```

That fixes both problems at once: it reproduces Java's overflow for large widths and removes the debug panic. The alternative for problem 1 is to reject `width > 2^30` at construction, which overlaps with #2473, but that would reject specs iceberg-java accepts.

### Willingness to contribute

I would be willing to contribute a fix for this bug with guidance from the Iceberg community.

---

**How this was found.** Apache DataFusion Comet is adding native kernels for Iceberg's Spark system functions (apache/datafusion-comet#5638). On a partitioned write the sort key comes from those kernels (Java semantics) while the partition values the clustered writer groups by come from iceberg-rust's transforms, and the clustered writer fails at runtime with `The input is not sorted! Cannot write to partition that was previously closed` when the two disagree. Cross-checking the two implementations over boundary inputs is what surfaced this.

_This report was drafted with LLM assistance (Claude Code); the divergences were verified against Iceberg Java on a JVM._

Contributor guide

Open the contributing guide

Research direction

Start in crates/iceberg/src/transform/truncate.rs by inspecting truncate_i32, truncate_i64, and truncate_decimal_i128. Reproduce the listed large-width and integer-minimum cases in debug and release builds, then compare the results with the Apache Iceberg Java TruncateUtil examples. Done means the kernels match Java for the reported boundary values without debug-build panics.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.