apache / apache/parquet-format

Clarify encoding schemes for boolean types

Open
#347 3 comments 0 reactions 0 assignees View on GitHub
Priority: Major Type: enhancement
Dominant language
Thrift
Stars
2.6k
Forks
508
Avg merge
2d 8h
Merged PRs (30d)
4

Description

In the Parquet format specification, under [the section for Plain encoding](https://github.com/apache/parquet-format/blob/master/Encodings.md#plain-plain--0), boolean is encoded using the deprecated bit-packed encoding. However, [the section for bit-packed encoding](https://github.com/apache/parquet-format/blob/master/Encodings.md#bit-packed-deprecated-bit_packed--4) specifies that it is only used for repetition/definition levels. This seems contradictory.

[The section for RLE/bit-packed hybrid encoding](https://github.com/apache/parquet-format/blob/master/Encodings.md#run-length-encoding--bit-packing-hybrid-rle--3) says "_Boolean values in data pages, as an alternative to PLAIN encoding_" - perhaps we should be specific and indicate this is only used for data page V2?

Also, implementation-wise, I saw parquet-cpp still encode boolean as plain 1-bit value while parquet-mr uses bit-packed encoding as described in the specification. Perhaps consolidation should be done for this.

**Reporter**: [Chao Sun](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=csun) / @sunchao
**Assignee**: [Chao Sun](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=csun) / @sunchao

**Note**: *This issue was originally created as [PARQUET-1249](https://issues.apache.org/jira/browse/PARQUET-1249). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start with Encodings.md, especially the Plain, bit-packed, and RLE/bit-packed hybrid sections linked in the issue. Compare the specification wording with the boolean encoding behavior in parquet-cpp and parquet-mr. Done means the boolean encoding rules are consistent, including any data page V2 distinction, and the specification no longer appears contradictory.

Written by the indexing model from the issue text.

Assessment

Domain
documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.