Skip Computing Nulls for Non-Nullable Parquet Columns
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Currently `DefinitionLevelDecoder` always computes a null mask if any parent of a column is nullable.
This is not only wasteful, but is also dubiously correct as it can result in StructArray that would fail the nullability checks:
Consider the case of
```
optional group l1 {
optional group l2 {
required INT32 leaf;
}
}
```
PrimitiveArrayDecoder will decode leaf with a null buffer, `l2` will then be decoded as a `StructArray` without a null buffer, with leaf as a child. This is technically ill-formed as the Field for leaf will state it isn't nullable, however, leaf has a null buffer
**Describe the solution you'd like**
`DefinitionLevelDecoder` should skip decoding a null buffer for non-nullable columns, some additional care may be necessary for dictionaries to ensure that the dictionary offsets are valid in isolation.
**Describe alternatives you've considered**
**Additional context**
Contributor guide
Research direction
Locate DefinitionLevelDecoder and trace how it creates null buffers for nested optional and required Parquet columns, including the PrimitiveArrayDecoder and StructArray path. Check the dictionary case mentioned in the issue; done means non-nullable columns no longer receive unnecessary null buffers while dictionary offsets remain valid and the nested example is well-formed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100