DataFusion INSERT fills omitted columns with NULL instead of the column's write-default
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 567
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 93
Description
### Apache Iceberg Rust version
0.6.0 (latest version)
### Describe the bug
The spec requires writers to use a column's `write-default` when the writer does not supply a value for the column. `IcebergTableProvider` does not implement DataFusion's `TableProvider::get_column_default`, so DataFusion's insert planner falls back to `NULL` for columns omitted from an `INSERT`. Rows written through DataFusion into a table whose schema declares `write-default` values silently get `NULL` instead of the default.
For comparison, iceberg-java delegates write-default application to the engine in the same way: its Spark integration exposes `write-default` as the engine-level column default (`TypeToSparkType`), and Spark materializes the value during INSERT planning. The DataFusion integration is the corresponding place in iceberg-rust.
### To Reproduce
```sql
-- table schema: id int (required), category string with write-default 'general'
INSERT INTO t (id) VALUES (1);
SELECT category FROM t; -- returns NULL, expected 'general'
```
### Expected behavior
`IcebergTableProvider` implements `get_column_default`, mapping each top-level column's `write-default` literal to a DataFusion expression, so the insert planner fills omitted columns with the declared default.
Part of #2411 (Default Values).
### Willingness to contribute
I can contribute a fix for this bug independently
Contributor guide
Research direction
Start at IcebergTableProvider and its TableProvider::get_column_default integration, then inspect the DataFusion INSERT planning path and the write-default schema values described in the issue. Reproduce the INSERT with an omitted category column, and consider the work done when the planner maps each top-level write-default literal to a DataFusion expression and the query returns 'general' instead of NULL.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100