apache / apache/parquet-java

Specification for RLEDictionary encoding is incorrect.

Open
#2,660 1 comment 0 reactions 0 assignees View on GitHub
Component: Parquet Priority: Major Type: bug
Dominant language
Java
Stars
3.1k
Forks
1.6k
Avg merge
3d 12h
Merged PRs (30d)
33

Description

The [spec for RLE Dictionary\|[https://github.com/apache/parquet-format/blob/master/Encodings.md#dictionary-encoding-plain_dictionary--2-and-rle_dictionary--8]] encoding says the "length of the encoded-data" is placed before the "encoded-data". Reproducing the first 3 lines here:

```

rle-bit-packed-hybrid:

length := length of the in bytes stored as 4 bytes little endian (unsigned int32)

encoded-data := \*

```

However, this is not true. Parquet-MR implementation does not encode the length in front of the data. It encodes bitWidth as 1 byte. See [implementation\|[https://github.com/apache/parquet-mr/blob/01a5d074829ad4cf4de1f662d54fe7bceb4bef63/parquet-column/src/main/java/org/apache/parquet/column/values/dictionary/DictionaryValuesWriter.java#L173]].

I'm proposing the spec be updated to state the above clearly.

see discussion here:

 

**Reporter**: [Balaji K](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=gamaken)

**Note**: *This issue was originally created as [PARQUET-2108](https://issues.apache.org/jira/browse/PARQUET-2108). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the RLE Dictionary section in Encodings.md and compare its wording with DictionaryValuesWriter.java at the linked implementation. Check the linked discussion for the intended interpretation, then update the specification so the documented byte layout matches the implementation and verify the surrounding encoding description remains consistent.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
documentation
Issue type
Documentation
Difficulty
1/5
Estimated time
Under an hour
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.