apache / apache/iceberg-rust

DataFusion INSERT fills omitted columns with NULL instead of the column's write-default

Open
#2,803 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

### Apache Iceberg Rust version

0.6.0 (latest version)

### Describe the bug

The spec requires writers to use a column's `write-default` when the writer does not supply a value for the column. `IcebergTableProvider` does not implement DataFusion's `TableProvider::get_column_default`, so DataFusion's insert planner falls back to `NULL` for columns omitted from an `INSERT`. Rows written through DataFusion into a table whose schema declares `write-default` values silently get `NULL` instead of the default.

For comparison, iceberg-java delegates write-default application to the engine in the same way: its Spark integration exposes `write-default` as the engine-level column default (`TypeToSparkType`), and Spark materializes the value during INSERT planning. The DataFusion integration is the corresponding place in iceberg-rust.

### To Reproduce

```sql
-- table schema: id int (required), category string with write-default 'general'
INSERT INTO t (id) VALUES (1);
SELECT category FROM t; -- returns NULL, expected 'general'
```

### Expected behavior

`IcebergTableProvider` implements `get_column_default`, mapping each top-level column's `write-default` literal to a DataFusion expression, so the insert planner fills omitted columns with the declared default.

Part of #2411 (Default Values).

### Willingness to contribute

I can contribute a fix for this bug independently

Contributor guide

Open the contributing guide

Research direction

Start at IcebergTableProvider and its TableProvider::get_column_default integration, then inspect the DataFusion INSERT planning path and the write-default schema values described in the issue. Reproduce the INSERT with an omitted category column, and consider the work done when the planner maps each top-level write-default literal to a DataFusion expression and the query returns 'general' instead of NULL.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.