Severe performance degradation after migrating to cramjam LZ4
- 主要語言
- Python
- 星號
- 1.4k
- 分支
- 269
- 平均合併
- 1 天 1 小時
- 30 天內合併 PR
- 6
描述
**Describe the bug**
In #960, when switching compression libraries, the default compression settings for LZ4 were also changed. In `python-lz4`, the default level was 0. Now it is 9.
We have seen an over 100% increase in producer latency in a production application due to this change. It happens to publish a relatively large binary payload. We are faced with a decision to either downgrade back to 0.10.0 or explicitly monkeypatch `codec.lz4_compress`.
Generally speaking, folks who are deliberately choosing to install lz4 when Standard Library natively supports others are opting for faster/weaker compression. From the [link](https://cwiki.apache.org/confluence/display/KAFKA/KIP-390%3A+Support+Compression+Level) included in the function documentation:

the choice of going from 0/1 -> 9 yields a 45% latency increase for a 1.2% improvement in compression ratio. I don't think folks are signing up for this trade-off.
**Expected behaviour**
Our "expected" behavior is that, since one cannot explicitly control compression settings and only `compression.type`, that settings are identical to they were when aiokafka directly used `python-lz4`. Could we consider reverting the default level to 0 to match the original behavior pre-0.11.0?
貢獻指南
評估
這個 Issue 還沒有評估資料。