Writing empty or all-null parquet column chunks creates an empty dictionary page
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Describe the bug**
Not sure if this is really a bug, but it is a bit unexpected and has lead to compatibility issues before (, now fixed). It also adds a little bit of metadata overhead, but that is probably not significant.
**To Reproduce**
Can be reproduced as a testcase in `parquet/src/column/writer/mod.rs`:
```rust
#[test]
fn test_column_writer_default_encoding_support_int32_empty() {
check_encoding_write_support::(
WriterVersion::PARQUET_1_0,
true,
&[],
Some(0), // dictionary page offset is set
&[Encoding::PLAIN, Encoding::RLE],
);
}
```
**Expected behavior**
To be discussed. There is actually a benefit to the current behavior when looking at `PageEncodingStats` (#7354), if one column chunk contains only nulls, and all other column chunks are dictionary encoded then the whole column could still be considered dictionary encoded.
**Additional context**
Contributor guide
Research direction
Start with the test case in parquet/src/column/writer/mod.rs and the check_encoding_write_support call for an empty Int32 column. Trace how the dictionary page offset and Encoding values are produced, then confirm the intended behavior for empty or all-null chunks, including the PageEncodingStats consideration. Done means the expected behavior is agreed and covered by a passing regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100