apache / apache/iceberg

Silent data nullification during MERGE when nullable struct contains non-nullable children

Open
#16,246 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Apache Iceberg version

1.5.2

### Query engine

Spark

### Please describe the bug 🐞

When performing a MERGE INTO operation on an Iceberg table where a nullable parent struct contains non-nullable child fields, the struct's non-nullable children silently become null — even though only other columns were updated. No error is raised.

Schema example:

Type: struct> (NULLABLE)

The parent field `Type` is nullable, but its child field `Type.Type` is non-nullable.

What happens:
During a MERGE operation that updates other columns (not the struct itself), the nullable parent struct's non-nullable child fields silently become null. No error is raised. Subsequent reads return null for `Type.Type` even though it's marked NOT NULL and was never modified by the MERGE.

Expected behavior:
Either:
1. Iceberg should preserve the existing struct data when the MERGE does not modify that column, OR
2. Iceberg should raise an error if a non-nullable field would become null (even transitively through a null parent)

Steps to reproduce:
1. Create an Iceberg table with a nullable struct column containing a non-nullable child field
2. Insert rows where the struct is populated (child field has a value)
3. Perform a MERGE INTO that updates other columns (not the struct itself) for some rows
4. Read back the table — the struct's non-nullable child field is now null despite not being touched by the MERGE

Additional context:
- This is not related to parent and child having the same name — tested with different names (Type.Type2) and the same issue occurs.
- Full table rewrites do not trigger the issue — only incremental MERGE operations.
- The issue may be related to how Iceberg reconciles Parquet file-level nullability with the Iceberg schema during partial updates.

Environment:
- Iceberg 1.5.2
- Spark 3.3
- Hadoop catalog type
- Storage: ABFSS (Azure Data Lake)

### Willingness to contribute

- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Research direction

Reproduce the issue with Spark 3.3 and Iceberg 1.5.2 using the nullable parent struct and non-nullable child described in the steps, then inspect schema reconciliation during a partial MERGE update. Done means the existing struct data is preserved when untouched, or the operation raises an error instead of silently nullifying the child field.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, hadoop, java, spark
Domain
data, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.