apache / apache/parquet-format
Clarify encoding schemes for boolean types
- Dominant language
- Thrift
- Stars
- 2.6k
- Forks
- 508
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 4
Description
In the Parquet format specification, under [the section for Plain encoding](https://github.com/apache/parquet-format/blob/master/Encodings.md#plain-plain--0), boolean is encoded using the deprecated bit-packed encoding. However, [the section for bit-packed encoding](https://github.com/apache/parquet-format/blob/master/Encodings.md#bit-packed-deprecated-bit_packed--4) specifies that it is only used for repetition/definition levels. This seems contradictory.
[The section for RLE/bit-packed hybrid encoding](https://github.com/apache/parquet-format/blob/master/Encodings.md#run-length-encoding--bit-packing-hybrid-rle--3) says "_Boolean values in data pages, as an alternative to PLAIN encoding_" - perhaps we should be specific and indicate this is only used for data page V2?
Also, implementation-wise, I saw parquet-cpp still encode boolean as plain 1-bit value while parquet-mr uses bit-packed encoding as described in the specification. Perhaps consolidation should be done for this.
**Reporter**: [Chao Sun](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=csun) / @sunchao
**Assignee**: [Chao Sun](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=csun) / @sunchao
**Note**: *This issue was originally created as [PARQUET-1249](https://issues.apache.org/jira/browse/PARQUET-1249). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
Contributor guide
Research direction
Start with Encodings.md, especially the Plain, bit-packed, and RLE/bit-packed hybrid sections linked in the issue. Compare the specification wording with the boolean encoding behavior in parquet-cpp and parquet-mr. Done means the boolean encoding rules are consistent, including any data page V2 distinction, and the specification no longer appears contradictory.
Written by the indexing model from the issue text.
Assessment
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100