[FEA] Add GZIP compression support to parquet writer
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Is your feature request related to a problem? Please describe.**
The parquet format in Apache Spark supports many compression codecs ([link](https://spark.apache.org/docs/2.4.3/sql-data-sources-parquet.html#configuration)), including: none, uncompressed, snappy, gzip, lzo, brotli, lz4, zstd.
cuDF has both internal implementation and an nvCOMP integration to provide compression and decompression codecs. For the parquet format, GZIP compression is [DEFLATE](https://developer.nvidia.com/blog/accelerating-load-times-for-directx-games-and-apps-with-gdeflate-for-directstorage/) plus a header. nvCOMP does not support the deflate version with this header, so the reader still uses the internal gzip decompression implementation. We don't have internal gzip compression implementation. To support GZIP in the PQ writer we would need to use nvCOMP GDEFLATE codec + write the header on our own.
**Describe the solution you'd like**
Add support for GZIP compressioning to the cuDF parquet writer by adding a header writing implementation and using nvCOMP deflate.
**Describe alternatives you've considered**
n/a
**Additional context**
Also see Spark-RAPIDS request here: https://github.com/NVIDIA/spark-rapids/issues/9718
Contributor guide
Assessment
This issue has not been assessed yet.