apache / apache/iceberg-rust

Type casting error when reading files persisted with old schema for complex type

Open
#2,617 0 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

### Apache Iceberg Rust version

None

### Describe the bug

When a table's struct (or nested list/map) column has gained fields over time via schema evolution, reading data files written under the older schema fails with an Arrow cast error such as Cast error: Casting from Utf8 to Struct(...). The record-batch transformer reconciles a file's nested children to the table schema by position within the struct rather than by Iceberg field id, so once a nested struct adds a field, the children no longer line up and a mismatched cast is attempted (e.g. casting a string child into a struct slot). Files are valid and readable by Iceberg-Java/Spark.

e.g. struct goes from a, c to a, b, c -> when reading old file with only a, c it tries to cast c to type of b

### To Reproduce

1. Create a table with a column s struct (plus other columns).
2. Write a data file.
3. Evolve the schema to s struct (add a nested field), keeping field ids stable.
4. Read the table (the older file still has s with only a).
5. The scan errors with Cast error: Casting from Utf8 to Struct(...).

### Expected behavior

Nested struct/list/map children are reconciled to the table schema by field id (recursively), and fields present in the table schema but absent from the file are materialized as typed NULLs — matching Iceberg's column-projection-by-id semantics. The read should succeed.

### Willingness to contribute

I can contribute a fix for this bug independently

Contributor guide

Open the contributing guide

Research direction

Start at the record-batch transformer, focusing on how nested struct, list, and map children are reconciled during reads. Reproduce the schema-evolution sequence with an older file and a newly added nested field. Done means old files read successfully, children are matched recursively by field id, and absent fields become typed NULLs without an Arrow cast error.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.