envoyproxy / envoyproxy/envoy

Add ISA-L to accelerate gzip compression in case of high network load

Open
#22,397 6 comments 0 reactions 0 assignees View on GitHub
area/compression area/extension enhancement no stalebot
Dominant language
C++
Stars
28.9k
Forks
5.6k
Avg merge
1d 22h
Merged PRs (30d)
430

Description

*Title*: *Add ISA-L to accelerate gzip compression in case of high network load*

*Description*:
[ISA-L](https://github.com/intel/isa-l) is a collection of optimized low-level functions targeting storage applications. The low-level functions of ISA-L are written in manual assembly code that many details are tuned to achieve optimal performance.

ISA-L uses efficient SIMD (Single Instruction, Multiple Data) instructions like AVX512 to maximize the use of CPU micro-architecture to accelerate the calculation process of compression algorithms. And ISA-L mainly optimized the implementation of CRC, DEFLATE and Huffman encoding for gzip compression algorithm.

We have performed a benchmark that test the compression performance under different data scales with ISA-L and Zlib, see figure in bellow. Specifically, we used [compressor_speed_test tool](https://github.com/envoyproxy/envoy/blob/main/test/extensions/filters/http/compressor/compressor_filter_speed_test.cc) to do benchmark, and averaged results of all compression levels for drawing the graph. And you can see all benchmark logs in [isa-l-benchmark](https://github.com/giantcroc/isa-l-benchmark).

![isa-l-benchmark](https://user-images.githubusercontent.com/20723282/180938953-c555cd54-eaa0-4cd7-9617-6bf504317196.png)

As shown in the figure above, we believe that ISA-L can help Envoy to improve gzip compression performance in case of **high network load**, and we are working on integrating ISA-L into Envoy.

[optional *Relevant Links*:]
[ISA-L Igzip: Improvements to a Fast Deflate](https://www.computer.org/csdl/proceedings-article/dcc/2017/07923748/12OmNyugyUz)
[ISA-L tool](https://www.intel.com/content/www/us/en/developer/tools/isa-l/overview.html)
[ISA-L videos](https://www.intel.cn/content/www/cn/zh/developer/videos/intelligent-storage-acceleration-library.html)
[ISA-L intro](https://s3.us-east-2.amazonaws.com/intel-builders/day_2_intelligent_storage_acceleration_library.pdf)
[ISA-L in DPDK](https://doc.dpdk.org/guides/compressdevs/isal.html)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.