apache / apache/iceberg-rust

Data files persisted without complex columns in the table schema hit "unexpected target column" on read

Open
#2,618 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

### Apache Iceberg Rust version

None

### Describe the bug

If we write a data file and then evolve our table's schema to add a complex type field like list, map, struct, proceeding to read the data file back will error out.

### To Reproduce

1. Create a table and write a data file (without column xs).
2. Evolve the schema to add xs list (or a map/struct column).
3. Read the table.
4. Reading the older file errors with unexpected target column type List(...).

### Expected behavior

A column present in the table schema but absent from a data file is filled with a typed NULL array of the correct (possibly nested) Arrow type via new_null_array, for nested types just as for primitives. The read should succeed with the added column all-NULL for older files.

### Willingness to contribute

None

Contributor guide

Open the contributing guide

Research direction

Start in the table-read path by searching for the "unexpected target column" error and uses of new_null_array. Reproduce the schema-evolution case with an older file missing a list, map, or struct column. Done means reading succeeds and the added complex column is a typed, all-NULL Arrow array, as it already should be for primitive columns.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.