Evaluate rebasing Hudi's default compression from Gzip to Zstd

Currently, having Gzip as a default we prioritize Compression/Storage cost at the expense of
* Compute (on the {+}write-path{+}): about *30%* of Compute burned during bulk-insert in local benchmarks on Amazon Reviews dataset is Gzip (see below) 
* Compute (on the {+}read-path{+}), as well as queries Latencies: queries scanning large datasets are likely to be compression-/CPU-bound (Gzip t/put is *3-4x* less than Snappy, Zstd, [EX|https://stackoverflow.com/a/56410326/3520840])

P.S Spark switched its default compression algorithm to Snappy [a while ago|https://github.com/apache/spark/pull/12256].

 

*EDIT*

We should actually evaluate putting in [zstd|https://engineering.fb.com/2016/08/31/core-data/smaller-and-faster-data-compression-with-zstandard/] instead of Snappy. It has compression ratios comparable to Gzip, while bringing in much better performance:

!image-2021-12-03-13-13-02-892.png!

[https://engineering.fb.com/2016/08/31/core-data/smaller-and-faster-data-compression-with-zstandard/]

 

 

 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-2928
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-3249
- Attachment(s):
- 03/Dec/21 21:03;alexey.kudinkin;Screen Shot 2021-12-03 at 12.36.13 PM.png;https://issues.apache.org/jira/secure/attachment/13036992/Screen+Shot+2021-12-03+at+12.36.13+PM.png
- 06/Dec/21 19:49;alexey.kudinkin;Screen Shot 2021-12-06 at 11.49.05 AM.png;https://issues.apache.org/jira/secure/attachment/13037052/Screen+Shot+2021-12-06+at+11.49.05+AM.png
- 03/Dec/21 21:13;alexey.kudinkin;image-2021-12-03-13-13-02-892.png;https://issues.apache.org/jira/secure/attachment/13036993/image-2021-12-03-13-13-02-892.png

---

## Comments

03/Dec/21 21:03;alexey.kudinkin;!Screen Shot 2021-12-03 at 12.36.13 PM.png!;;;

---

06/Dec/21 19:50;alexey.kudinkin;Running a benchmark upon small subset of the Amazon Reviews dataset we're able to see considerable improvement in bulk-insert times: bulk-insert was up to *40%* faster, while it had very similar footprint in the storage.

!Screen Shot 2021-12-06 at 11.49.05 AM.png|width=935,height=644!;;;

---

14/Dec/21 01:06;alexey.kudinkin;Unfortunately, the switching to Zstd might required a little more grinding than initially anticipated:

Current Parquet version (1.10.1, being handed down by Spark 2.4.4) only supports `ZstdCompressionCodec` as provided by "hadoop-common", which in turn requires it to be built with Native Libraries support (including compression codecs, etc). It only supports Linux/*nix.

Therefore if we're planning on supporting Spark 2.x we have following options: 
# Implement our own version of `ZstdCompressionCodec` leveraging either [zstd-jni|https://github.com/luben/zstd-jni] (used by Spark internally) or airlift-aircompressor (claims to be faster than JNI impl).
# Switch to `zstd` being default setting only for Spark 3 environments.

 ;;;

---

12/Jan/22 00:45;alexey.kudinkin;Unfortunately we won't be able to support Zstd w/o herculean effort of hacking around Parquet implementation as it's not unfortunately modularized well-enough to support outside extensions.

 

The only sensible way at this point seem to be waiting for Spark/Parquet upgrade to 1.12.;;;

---

03/Feb/22 17:29;alexey.kudinkin;Uber's example of leveraging Zstd in lieu of Gzip

https://eng.uber.com/cost-efficiency-big-data/;;;

No source file or test is named. Review the current Gzip default and the Parquet 1.10.1/Spark 2.4.4 compatibility constraints described in the thread, then reproduce the Amazon Reviews bulk-insert benchmark comparing Zstd and Gzip. Done means a documented decision and an implementable compatibility path, including the Linux native-library limitation.

Written by the indexing model from the issue text.