OPEN_STRUCT: declared child defaultNullValue is honored only for keys absent from the whole segment
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 55m
- Merged PRs (30d)
- 182
Description
An OPEN_STRUCT child field spec can declare a custom `defaultNullValue`, but only one of the four per-key storage layouts honors it. With null handling disabled, the same query can therefore return different values for the same key depending only on which segment answers it.
### Current behavior
A document that does not carry the key reads as:
| Layout | Effective default | Resolved by |
|---|---|---|
| Key absent from the whole segment | the declared child spec's `defaultNullValue` | `OpenStructDataSource#getValueFieldSpec` |
| Materialized (dense) key | standard dimension default of the stored type | `OpenStructColumnSplitter#writeDenseKeyColumn` |
| Sparse key | standard dimension default of the stored type | `SparseKeyDataSource` |
| Consuming (mutable) key | standard dimension default of the stored type | `MutableOpenStructIndex#allocateKeyColumn` |
So for a declared STRING child with `"defaultNullValue": "N/A"`, a segment in which the key never occurs reads `N/A` for every row, while a segment in which the key occurs at least once reads `null` for every other row.
### Desired
Pick one effective-default rule and apply it across all four layouts. The natural choice is the declared child field spec, which `OpenStructDataSource#getValueFieldSpec` already returns, since it is the only rule that reflects what was configured.
The sealed and consuming paths are coupled: `MutableOpenStructIndex` deliberately mirrors `OpenStructColumnSplitter` so that a document's resolved value does not change when a consuming segment is sealed. They have to move together.
`MapFilterOperator#trySparseJsonIndex` already refuses the sparse JSON fast path for the key's declared default, so it needs no change.
### Code sites
- `OpenStructColumnSplitter#writeDenseKeyColumn`
- `MutableOpenStructIndex#allocateKeyColumn`
- `SparseKeyDataSource`
Each carries a TODO pointing at this issue.
Contributor guide
Research direction
Start with the TODOs in OpenStructColumnSplitter#writeDenseKeyColumn, MutableOpenStructIndex#allocateKeyColumn, and SparseKeyDataSource, then read OpenStructDataSource#getValueFieldSpec to understand the declared child default. Compare the sealed and consuming paths, including their coupling, across the four layouts. Done means every layout uses the declared defaultNullValue consistently while MapFilterOperator#trySparseJsonIndex remains unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100