NVIDIA / NVIDIA/cudf

[FEA] Confirm correct and efficient dictionary processing in libcudf

Open
#15,199 6 comments 0 reactions 0 assignees View on GitHub
cuIO feature request libcudf
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Is your feature request related to a problem? Please describe.**
Using a parquet reader option, we could allow the user to specify columns that they would like to receive as dictionary-encoded in the output table. For the specified columns, the Parquet reader would transcode multiple Parquet dictionary-encoded column chunks into an Arrow dictionary-encoded column.

**Describe the solution you'd like**
### Part 1 - Confirm correct and efficient dictionary processing in libcudf ###
1. Add benchmarks for dictionary `encode` and `decode` with axes including data type, cardinality and row count. Add checks that data is correctly round-tripped through dictionary encoding and decoding.
2. Expand unit testing when using dictionary types for reductions, join keys, aggregation keys, aggregation values and other operations. Include string and numeric types as dictionary values. Please note that although libcudf can represent dictionaries of lists (needs to be checked), in Parquet only leaf values can be dictionary-encoded.
3. Expand benchmarks for dictionary operations. As of 24.04 we only have a [dictionary reduction](https://github.com/rapidsai/cudf/blob/branch-24.04/cpp/benchmarks/reduction/dictionary.cpp) benchmarks on `int32` and `float` value types. Benchmarks should include strings data type and axes for varying cardinality and row count.
4. Consider signed int for index type. Revisit the int types that can be used as indices. Revisit compatibility differences between libcudf dictionary and Arrow dictionary.
5. Consider dropping the sorted key requirement for improved python compatibility. We use natural order of index today and we could add a mapping layer to indexes to stop constraining the indices.

### Part 2 - Parquet-to-Arrow dictionary transcoding ✅ ###
1. Estimate the performance of transcoding Parquet dictionary-encoded column chunks into arrow dictionary-encoded columns. Each Parquet dictionary-encoded column chunk with begins with a dictionary page. To create an Arrow-compliant dictionary column, we need to merge the values from the dictionary page in each column chunk into a single set of values for the arrow dictionary-encoded column. Then to generate the indices data, we need to re-map the indices from each column chunk against the indices in the combined values.
2. Please note that [PLAIN_DICTIONARY](https://parquet.apache.org/docs/file-format/data-pages/encodings/#dictionary-encoding-plain_dictionary--2-and-rle_dictionary--8) encoding is deprecated in Parquet 2.0. To support the new default [RLE_DICTIONARY](https://parquet.apache.org/docs/file-format/data-pages/encodings/#dictionary-encoding-plain_dictionary--2-and-rle_dictionary--8), we will need to add a conversion step from Parquet bit-packed indices into Arrow fixed-width indices.
3. The parquet format allows different encodings for each column chunk within a column. In the case of dictionaries, the Parquet specification describes cases where PLAIN encoding will be mixed with DICTIONARY encoding, "If the dictionary grows too big, whether in size or number of distinct values, the encoding will fall back to the plain encoding". To support this case we would need to add special handling.

Completed by https://github.com/NVIDIA/cudf/pulls?q=23710+22532

### Part 3 - Introduce run-end encoded type in libcudf, and then add Parquet-to-Arrow run-length/run-end transcoding
The Parquet format supports a [run-length encoding / bit-packing hybrid](https://parquet.apache.org/docs/file-format/data-pages/encodings/#run-length-encoding--bit-packing-hybrid-rle--3) and this could be transcoded into a [run-end encoded](https://arrow.apache.org/docs/format/Columnar.html#run-end-encoded-layout) Arrow type. To begin this project, we need to add run-end encoding as a new type to libcudf, introduce decode and encode functions, confirm correctness across libcudf APIs and audit for performance hotspots. A run-end encoded type in libcudf would allow us to support "constant" or "scalar" columns as requested in #15308. If libcudf supported a run-end encoded type, transcoding into this type from Parquet run-length encoded data would not be a zero-copy operation and would require converting the Parquet bit-packed "lengths" to Arrow fixed-width "ends".

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.