apache / apache/iceberg

Decimal truncate transform can produce values that exceed the source precision

Open
#17,027 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 16h
Merged PRs (30d)
129

Description

### Apache Iceberg version

1.9.0

### Query engine

Spark

### Please describe the bug 🐞

This is a follow-up to #12915, which was automatically closed as stale without a resolution.

The decimal truncate transform can increase the precision of a negative boundary value beyond the precision of the source decimal type. For example, truncating a valid `decimal(7,3)` value with width 10 maps:

```
-9999.999 -> -10000.000
```

The input has precision 7, while the transformed value requires precision 8. A valid source value can therefore produce a partition value that cannot be represented by the transform's declared decimal type.

A minimal Spark SQL reproduction is:

```sql
CREATE TABLE sample (a decimal(7,3))
USING iceberg
PARTITIONED BY (truncate(10, a));

INSERT INTO sample VALUES (-9999.999);
```

The write fails when Iceberg serializes the transformed partition value:

```
Caused by: java.lang.IllegalArgumentException:
Cannot write value as decimal(7,3), too large: -10000.000
at org.apache.iceberg.util.DecimalUtil.toReusedFixLengthBytes(DecimalUtil.java:39)
at org.apache.iceberg.avro.ValueWriters$DecimalWriter.write(ValueWriters.java:345)
```

The same behavior occurs for other precisions and scales when a negative value near the minimum boundary is rounded down by the truncate transform, such as the `decimal(5,4)` example in #12915.

Iceberg should handle this boundary case without allowing an unrepresentable transformed value to reach the metadata writer. This may require defining how decimal truncate behaves when the mathematical result exceeds the source precision.

### Willingness to contribute

- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Research direction

Start with the decimal truncate implementation and the serialization paths shown in DecimalUtil.java and ValueWriters.java. Run the supplied Spark SQL reproduction and add boundary-focused coverage for negative decimal values; done means valid source values no longer produce an unrepresentable transformed partition value or fail during metadata serialization.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.