Type casting error when reading files persisted with old schema for complex type
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 567
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 93
Description
### Apache Iceberg Rust version
None
### Describe the bug
When a table's struct (or nested list/map) column has gained fields over time via schema evolution, reading data files written under the older schema fails with an Arrow cast error such as Cast error: Casting from Utf8 to Struct(...). The record-batch transformer reconciles a file's nested children to the table schema by position within the struct rather than by Iceberg field id, so once a nested struct adds a field, the children no longer line up and a mismatched cast is attempted (e.g. casting a string child into a struct slot). Files are valid and readable by Iceberg-Java/Spark.
e.g. struct goes from a, c to a, b, c -> when reading old file with only a, c it tries to cast c to type of b
### To Reproduce
1. Create a table with a column s struct (plus other columns).
2. Write a data file.
3. Evolve the schema to s struct (add a nested field), keeping field ids stable.
4. Read the table (the older file still has s with only a).
5. The scan errors with Cast error: Casting from Utf8 to Struct(...).
### Expected behavior
Nested struct/list/map children are reconciled to the table schema by field id (recursively), and fields present in the table schema but absent from the file are materialized as typed NULLs — matching Iceberg's column-projection-by-id semantics. The read should succeed.
### Willingness to contribute
I can contribute a fix for this bug independently
Contributor guide
Research direction
Start at the record-batch transformer, focusing on how nested struct, list, and map children are reconciled during reads. Reproduce the schema-evolution sequence with an older file and a newly added nested field. Done means old files read successfully, children are matched recursively by field id, and absent fields become typed NULLs without an Arrow cast error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100