NVIDIA / NVIDIA/cudf

[FEA] Add GZIP compression support to parquet writer

Open
#14,509 2 comments 2 reactions 0 assignees View on GitHub
0 - Backlog cuIO feature request libcudf Spark
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Is your feature request related to a problem? Please describe.**
The parquet format in Apache Spark supports many compression codecs ([link](https://spark.apache.org/docs/2.4.3/sql-data-sources-parquet.html#configuration)), including: none, uncompressed, snappy, gzip, lzo, brotli, lz4, zstd.

cuDF has both internal implementation and an nvCOMP integration to provide compression and decompression codecs. For the parquet format, GZIP compression is [DEFLATE](https://developer.nvidia.com/blog/accelerating-load-times-for-directx-games-and-apps-with-gdeflate-for-directstorage/) plus a header. nvCOMP does not support the deflate version with this header, so the reader still uses the internal gzip decompression implementation. We don't have internal gzip compression implementation. To support GZIP in the PQ writer we would need to use nvCOMP GDEFLATE codec + write the header on our own.

**Describe the solution you'd like**
Add support for GZIP compressioning to the cuDF parquet writer by adding a header writing implementation and using nvCOMP deflate.

**Describe alternatives you've considered**
n/a

**Additional context**
Also see Spark-RAPIDS request here: https://github.com/NVIDIA/spark-rapids/issues/9718

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.