aio-libs / aio-libs/aiokafka

Severe performance degradation after migrating to cramjam LZ4

未關閉
#1,028 11 則留言 1 個 reaction 已指派 0 人 在 GitHub 檢視
主要語言
Python
星號
1.4k
分支
269
平均合併
1 天 1 小時
30 天內合併 PR
6

描述

**Describe the bug**

In #960, when switching compression libraries, the default compression settings for LZ4 were also changed. In `python-lz4`, the default level was 0. Now it is 9.

We have seen an over 100% increase in producer latency in a production application due to this change. It happens to publish a relatively large binary payload. We are faced with a decision to either downgrade back to 0.10.0 or explicitly monkeypatch `codec.lz4_compress`.

Generally speaking, folks who are deliberately choosing to install lz4 when Standard Library natively supports others are opting for faster/weaker compression. From the [link](https://cwiki.apache.org/confluence/display/KAFKA/KIP-390%3A+Support+Compression+Level) included in the function documentation:

![image](https://github.com/aio-libs/aiokafka/assets/3999513/0349598f-9539-48a2-8a4e-e8f3afdb5ae6)

the choice of going from 0/1 -> 9 yields a 45% latency increase for a 1.2% improvement in compression ratio. I don't think folks are signing up for this trade-off.

**Expected behaviour**

Our "expected" behavior is that, since one cannot explicitly control compression settings and only `compression.type`, that settings are identical to they were when aiokafka directly used `python-lz4`. Could we consider reverting the default level to 0 to match the original behavior pre-0.11.0?

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。