apache / apache/pinot

OPEN_STRUCT: declared child defaultNullValue is honored only for keys absent from the whole segment

Open
#19,466 0 comments 0 reactions 0 assignees View on GitHub
bug null support query
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

An OPEN_STRUCT child field spec can declare a custom `defaultNullValue`, but only one of the four per-key storage layouts honors it. With null handling disabled, the same query can therefore return different values for the same key depending only on which segment answers it.

### Current behavior

A document that does not carry the key reads as:

| Layout | Effective default | Resolved by |
|---|---|---|
| Key absent from the whole segment | the declared child spec's `defaultNullValue` | `OpenStructDataSource#getValueFieldSpec` |
| Materialized (dense) key | standard dimension default of the stored type | `OpenStructColumnSplitter#writeDenseKeyColumn` |
| Sparse key | standard dimension default of the stored type | `SparseKeyDataSource` |
| Consuming (mutable) key | standard dimension default of the stored type | `MutableOpenStructIndex#allocateKeyColumn` |

So for a declared STRING child with `"defaultNullValue": "N/A"`, a segment in which the key never occurs reads `N/A` for every row, while a segment in which the key occurs at least once reads `null` for every other row.

### Desired

Pick one effective-default rule and apply it across all four layouts. The natural choice is the declared child field spec, which `OpenStructDataSource#getValueFieldSpec` already returns, since it is the only rule that reflects what was configured.

The sealed and consuming paths are coupled: `MutableOpenStructIndex` deliberately mirrors `OpenStructColumnSplitter` so that a document's resolved value does not change when a consuming segment is sealed. They have to move together.

`MapFilterOperator#trySparseJsonIndex` already refuses the sparse JSON fast path for the key's declared default, so it needs no change.

### Code sites

- `OpenStructColumnSplitter#writeDenseKeyColumn`
- `MutableOpenStructIndex#allocateKeyColumn`
- `SparseKeyDataSource`

Each carries a TODO pointing at this issue.

Contributor guide

Open the contributing guide

Research direction

Start with the TODOs in OpenStructColumnSplitter#writeDenseKeyColumn, MutableOpenStructIndex#allocateKeyColumn, and SparseKeyDataSource, then read OpenStructDataSource#getValueFieldSpec to understand the declared child default. Compare the sealed and consuming paths, including their coupling, across the four layouts. Done means every layout uses the declared defaultNullValue consistently while MapFilterOperator#trySparseJsonIndex remains unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.