apache / apache/iceberg

`write-default` null representation issue

Open
#15,932 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Apache Iceberg version

None

### Query engine

None

### Please describe the bug 🐞

TL;DR:
It's not always possible to set the DEFAULT value for newly written rows to NULL, without also altering what the DEFAULT value for existing rows should be.

Because of the backwards-compatibility of `write-default`, it will default to `initial-default` if it's not present.
To represent `NULL`, the default has to be omitted.

`null` seems to not be an accepted state for either `initial-default` or `write-default`.
The spec doesn't say they're nullable, and the iceberg-rest-fixture Catalog will return 500 if it reads a `initial-default` or `write-default` that is `null`

That means it's impossible to set the `write-default` to NULL, unless `initial-default` is also NULL (omitted)

Table to illustrate the problem:
```
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ initial-default β”‚ write-default β”‚ result (new rows) β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ 5 β”‚ - β”‚ 5 β”‚
β”‚ - β”‚ 21 β”‚ 21 β”‚
β”‚ 5 β”‚ 21 β”‚ 21 β”‚
β”‚ - β”‚ - β”‚ NULL β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
```

`write-default` is omitted (`-`), but because `initial-default` is non-null, the result won't be NULL

### Willingness to contribute

- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [x] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Research direction

Start by reading the Iceberg specification sections defining initial-default and write-default, then reproduce the behavior with the iceberg-rest-fixture Catalog. Determine how a NULL write-default should be represented independently of initial-default; done when the representation is accepted without the Catalog returning a 500 error.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.