Data files persisted without complex columns in the table schema hit "unexpected target column" on read
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 567
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 93
Description
### Apache Iceberg Rust version
None
### Describe the bug
If we write a data file and then evolve our table's schema to add a complex type field like list, map, struct, proceeding to read the data file back will error out.
### To Reproduce
1. Create a table and write a data file (without column xs).
2. Evolve the schema to add xs list (or a map/struct column).
3. Read the table.
4. Reading the older file errors with unexpected target column type List(...).
### Expected behavior
A column present in the table schema but absent from a data file is filled with a typed NULL array of the correct (possibly nested) Arrow type via new_null_array, for nested types just as for primitives. The read should succeed with the added column all-NULL for older files.
### Willingness to contribute
None
Contributor guide
Research direction
Start in the table-read path by searching for the "unexpected target column" error and uses of new_null_array. Reproduce the schema-evolution case with an older file missing a list, map, or struct column. Done means reading succeeds and the added complex column is a typed, all-NULL Arrow array, as it already should be for primitive columns.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100