`write-default` null representation issue
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Apache Iceberg version
None
### Query engine
None
### Please describe the bug π
TL;DR:
It's not always possible to set the DEFAULT value for newly written rows to NULL, without also altering what the DEFAULT value for existing rows should be.
Because of the backwards-compatibility of `write-default`, it will default to `initial-default` if it's not present.
To represent `NULL`, the default has to be omitted.
`null` seems to not be an accepted state for either `initial-default` or `write-default`.
The spec doesn't say they're nullable, and the iceberg-rest-fixture Catalog will return 500 if it reads a `initial-default` or `write-default` that is `null`
That means it's impossible to set the `write-default` to NULL, unless `initial-default` is also NULL (omitted)
Table to illustrate the problem:
```
ββββββββββββββββββββ¬βββββββββββββββββ¬βββββββββββββββββββββ
β initial-default β write-default β result (new rows) β
ββββββββββββββββββββΌβββββββββββββββββΌβββββββββββββββββββββ€
β 5 β - β 5 β
β - β 21 β 21 β
β 5 β 21 β 21 β
β - β - β NULL β
ββββββββββββββββββββ΄βββββββββββββββββ΄βββββββββββββββββββββ
```
`write-default` is omitted (`-`), but because `initial-default` is non-null, the result won't be NULL
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [x] I cannot contribute a fix for this bug at this time
Contributor guide
Research direction
Start by reading the Iceberg specification sections defining initial-default and write-default, then reproduce the behavior with the iceberg-rest-fixture Catalog. Determine how a NULL write-default should be represented independently of initial-default; done when the representation is accepted without the Catalog returning a 500 error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100