apache / apache/arrow-rs

Skip Computing Nulls for Non-Nullable Parquet Columns

Open
#4,252 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

Currently `DefinitionLevelDecoder` always computes a null mask if any parent of a column is nullable.

This is not only wasteful, but is also dubiously correct as it can result in StructArray that would fail the nullability checks:

Consider the case of

```
optional group l1 {
optional group l2 {
required INT32 leaf;
}
}
```

PrimitiveArrayDecoder will decode leaf with a null buffer, `l2` will then be decoded as a `StructArray` without a null buffer, with leaf as a child. This is technically ill-formed as the Field for leaf will state it isn't nullable, however, leaf has a null buffer

**Describe the solution you'd like**

`DefinitionLevelDecoder` should skip decoding a null buffer for non-nullable columns, some additional care may be necessary for dictionaries to ensure that the dictionary offsets are valid in isolation.

**Describe alternatives you've considered**

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Locate DefinitionLevelDecoder and trace how it creates null buffers for nested optional and required Parquet columns, including the PrimitiveArrayDecoder and StructArray path. Check the dictionary case mentioned in the issue; done means non-nullable columns no longer receive unnecessary null buffers while dictionary offsets remain valid and the nested example is well-formed.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.