apache / apache/parquet-format
[Format] Encoding spec incorrect for dictionary fallback
- Dominant language
- Thrift
- Stars
- 2.6k
- Forks
- 508
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 4
Description
The spec for DICTIONARY_ENCODING states that:
> If the dictionary grows too big, whether in size or number of distinct values, the encoding will fall back to the plain encoding.
https://github.com/apache/parquet-format/blob/master/Encodings.md#dictionary-encoding-plain_dictionary--2-and-rle_dictionary--8
However, the parquet-mr implementation was deliberately changed to a different fallback mechanism in https://issues.apache.org/jira/browse/PARQUET-52
I'm assuming the parquet-mr implementation is authoritative here. But then the spec is incorrect and should be fixed to reflect expected behavior.
**Reporter**: [Antoine Pitrou](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=apitrou) / @pitrou
**Note**: *This issue was originally created as [PARQUET-2221](https://issues.apache.org/jira/browse/PARQUET-2221). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
Contributor guide
Research direction
Start with the dictionary encoding section in Encodings.md and compare its fallback wording with the parquet-mr behavior described in PARQUET-52. Done means the specification accurately reflects the authoritative fallback mechanism and no longer incorrectly states that encoding falls back to plain encoding.
Written by the indexing model from the issue text.
Assessment
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 70/100