apache / apache/iceberg

Arrow: Vectorized reads of decimal columns with default values fail with IllegalArgumentException

Open
#16,502 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Apache Iceberg version

main (development)

### Query engine

Spark

### Please describe the bug 🐞

## Issue Summary

When the vectorized Arrow reader is used to read a v3 Iceberg table that has a `decimal` column carrying an `initialDefault` or `writeDefault`, vector allocation fails with:

```
java.lang.IllegalArgumentException: Cannot cast default value to FIXED[9]: 12345.6789
at org.apache.iceberg.types.Types$NestedField.castDefault(Types.java:892)
at org.apache.iceberg.types.Types$NestedField.(Types.java:881)
at org.apache.iceberg.types.Types$NestedField$Builder.build(Types.java:850)
at org.apache.iceberg.arrow.vectorized.VectorizedArrowReader.getPhysicalType(VectorizedArrowReader.java:255)
at org.apache.iceberg.arrow.vectorized.VectorizedArrowReader.allocateFieldVector(VectorizedArrowReader.java:228)
at org.apache.iceberg.arrow.vectorized.VectorizedArrowReader.read(VectorizedArrowReader.java:151)
```

The message varies with the underlying Parquet physical encoding:
- `FIXED_LEN_BYTE_ARRAY`-backed decimal → `Cannot cast default value to fixed[N]: `

Same read path with vectorization disabled has no errors:

```
spark.sql.iceberg.vectorization.enabled=false
```

## Repro

1. Create a v3 Iceberg table with a decimal column that has a default value:

```sql
CREATE TABLE local.db.t (
id INT,
amount DECIMAL(5, 2) DEFAULT 0.00
) USING iceberg TBLPROPERTIES ('format-version' = '3');

INSERT INTO local.db.t VALUES (1, 1.23), (2, 4.56), (3, 7.89);
```

2. Read with vectorization enabled (the default):

```sql
SET spark.sql.iceberg.vectorization.enabled=true;
SELECT * FROM local.db.t;
```

The query fails with the stack trace above. The failure is deterministic only when the column is not dictionary-encoded; with dictionary encoding, allocation goes through `allocateDictEncodedVector` and bypasses the buggy path, so small/highly-repetitive data sets may appear to read successfully.

## Root cause

`VectorizedArrowReader#getPhysicalType` rewrites a decimal Iceberg field to its underlying physical type (`int` / `long` / `fixed[N]`) so the right Arrow vector class can be allocated:

```java
physicalType = Types.NestedField.from(logicalType).ofType(type).build();
```

`Types.NestedField.Builder.from(field)` copies the field's `initialDefault` and `writeDefault` onto the builder. `NestedField`'s constructor then calls `castDefault(literal, type)` against the new physical type — for a decimal default this delegates to `DecimalLiteral.to(LongType | IntegerType | FixedType)`, which is undefined and returns `null`, tripping the `Preconditions.checkArgument` in `castDefault`.

Conceptually, the defaults belong to the logical (decimal) view of the column and should not flow to the physical representation — the physical type is an internal detail used only to size the Arrow vector. The non-vectorized readers (`BaseParquetReaders`, `SparkParquetReaders`, `FlinkParquetReaders`) all apply defaults at the logical-type layer and are unaffected.

Proposed PR for the fix: https://github.com/apache/iceberg/pull/16501

### Willingness to contribute

- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Research direction

Start with org.apache.iceberg.arrow.vectorized.VectorizedArrowReader#getPhysicalType and the Types.NestedField stack-trace locations, then reproduce the SQL example with Spark vectorization enabled. Done means the v3 table with a decimal default can be read through the vectorized path without the IllegalArgumentException; a pull request is already linked to this issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.