apache / apache/parquet-format

[Format] Encoding spec incorrect for dictionary fallback

Open Beginner friendly
#404 4 comments 0 reactions 0 assignees View on GitHub
Priority: Critical Type: bug
Dominant language
Thrift
Stars
2.6k
Forks
508
Avg merge
2d 8h
Merged PRs (30d)
4

Description

The spec for DICTIONARY_ENCODING states that:

> If the dictionary grows too big, whether in size or number of distinct values, the encoding will fall back to the plain encoding.

https://github.com/apache/parquet-format/blob/master/Encodings.md#dictionary-encoding-plain_dictionary--2-and-rle_dictionary--8

However, the parquet-mr implementation was deliberately changed to a different fallback mechanism in https://issues.apache.org/jira/browse/PARQUET-52

I'm assuming the parquet-mr implementation is authoritative here. But then the spec is incorrect and should be fixed to reflect expected behavior.

**Reporter**: [Antoine Pitrou](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=apitrou) / @pitrou

**Note**: *This issue was originally created as [PARQUET-2221](https://issues.apache.org/jira/browse/PARQUET-2221). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start with the dictionary encoding section in Encodings.md and compare its fallback wording with the parquet-mr behavior described in PARQUET-52. Done means the specification accurately reflects the authoritative fallback mechanism and no longer incorrectly states that encoding falls back to plain encoding.

Written by the indexing model from the issue text.

Assessment

Domain
documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.