NVIDIA / NVIDIA/cudf

[FEA] parquet: rle_stream for dictionary pages

Open
#14,950 2 comments 0 reactions 1 assignee Assigned to @pmattione-nvidia View on GitHub
cuIO feature request Performance Spark
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

I've been looking at the `rle_stream` class in order to decode dictionary streams in addition to repetition streams in the parquet decoder. This is a component of the work that @nvdbaranec has done here https://github.com/rapidsai/cudf/pull/13622, where we'd like to separate out at least a "fixed width" and a "fixed width dictionary encoded" pair of kernels.

With the changes in `rle_stream`, the core of the logic is able to use more threads for the RLE stream decoder. Specifically, a first warp is in charge of generating a set of runs, and other warps are able to take each one of the runs and decode them in parallel. As part of the micro kernel work, we feel that focusing on `rle_stream` decoder and its effects on `gpuComputeStringPageBounds`, `gpuComputePageSizes` and the use in the new fixed kernels, is a good first step to get the micro kernel work merged.

This issue then is to get a new `rle_stream` into cuDF that can handle both repetition AND dictionary streams, and show that the performance impact is same or better than what we have now. We hope that having this decoder will help centralize code, helping cleanup the parquet code base.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.