NVIDIA / NVIDIA/cudf

[FEA][JNI] Consider defaulting parquet dictionary encoding policy to ALWAYS

Open
#15,580 0 comments 0 reactions 0 assignees View on GitHub
cuIO feature request Java Spark
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

This PR is going to set the cuDF dictionary encoding policy for parquet to ADAPTIVE (https://github.com/rapidsai/cudf/pull/15570)

This is to get around an issue in nvcomp zstd https://github.com/rapidsai/cudf/issues/15501, where too large pages are getting created and is causing zstd to not compress larger dictionary pages.

For now we can pick ALWAYS in order to retain the current behavior for Spark. We should consider the impact of this setting for different compression, especially zstd (reading and writing).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.