apache / apache/arrow-rs

Writing empty or all-null parquet column chunks creates an empty dictionary page

Open
#7,356 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Describe the bug**

Not sure if this is really a bug, but it is a bit unexpected and has lead to compatibility issues before (, now fixed). It also adds a little bit of metadata overhead, but that is probably not significant.

**To Reproduce**

Can be reproduced as a testcase in `parquet/src/column/writer/mod.rs`:
```rust
#[test]
fn test_column_writer_default_encoding_support_int32_empty() {
check_encoding_write_support::(
WriterVersion::PARQUET_1_0,
true,
&[],
Some(0), // dictionary page offset is set
&[Encoding::PLAIN, Encoding::RLE],
);
}
```

**Expected behavior**

To be discussed. There is actually a benefit to the current behavior when looking at `PageEncodingStats` (#7354), if one column chunk contains only nulls, and all other column chunks are dictionary encoded then the whole column could still be considered dictionary encoded.

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start with the test case in parquet/src/column/writer/mod.rs and the check_encoding_write_support call for an empty Int32 column. Trace how the dictionary page offset and Encoding values are produced, then confirm the intended behavior for empty or all-null chunks, including the PageEncodingStats consideration. Done means the expected behavior is agreed and covered by a passing regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.