asyncio_websockets tests the implementation of `zlib`, not of Python

I found out that the asyncio_websockets benchmark spends ~87% of runtime in zlib (i.e. in the shared library libz.so), or whatever compression library is the default on the system-under-test.

In other words, asyncio_websockets tests the implementation of compression/decompression algorithms rather than anything to do with the Python interpreter or the websockets Python module. Websockets indeed enables compression by default: https://websockets.readthedocs.io/en/stable/topics/compression.html, excerpt from that official documentation:

connect() and serve() enable compression by default because the reduction in network bandwidth is usually worth the additional memory and CPU cost.

Problem

I believe this benchmark may not be measuring what's intended in its current form. For example, a replacement of zlib with zlib-ng or zlib-rs (which are newer drop-in replacement of zlib) may significantly affect the performance score of this benchmark, even though nothing changed in Python and/or websockets implementations. It is hard to root cause such performance modification, without knowing this detail about the asyncio_websockets benchmark.

It is also used in e.g. Phoronix testing, which may lead to unexpected conclusions for readers who aren't aware of this detail. Example: https://www.phoronix.com/review/cachyos-ubuntu-2510-f43/5.

Solutions

I see the following solutions:

1. Clearly document this behavior in https://pyperformance.readthedocs.io/benchmarks.html (in fact, there is no mention of websockets benchmark at all).
2. Remove this benchmark, since the workload is dominated by native zlib rather than Python or websockets logic.
3. Modify this benchmark to disable compression, as described here: https://websockets.readthedocs.io/en/stable/topics/compression.html#configuring-compression

I'd lean towards option 3 (disabling compression) as it preserves the benchmark's intent while removing the zlib dependency from results. I'm happy to submit a PR if the maintainers agree.

Reproducing

I ran it with a Amazon Linux 2023 docker container (OS distro similar to Fedora):

docker run --rm -it amazonlinux:2023 /bin/bash # -->

dnf install -y pip perf dnf-utils
dnf debuginfo-install zlib python3.9
python3 -m pip install pyperformance
python3 -m pip install websockets==11.0.3 pyperf==2.6.3

perf record -g --call-graph dwarf -- \
  python3 -u /usr/local/lib/python3.9/site-packages/pyperformance/data-files/benchmarks/bm_asyncio_websockets/run_benchmark.py

After running the benchmark, we can examine the resulting perf.data file:

$ perf report --hierarchy
...
-   99.93%        python3
   -   87.22%        libz.so.1.2.11
      +   40.11%        [.] inflate_fast
      +   36.28%        [.] deflate_slow
      ...
   +    5.20%        libpython3.9.so.1.0
   +    1.60%        libc.so.6
   ...

P.S. Thank you for maintaining the pyperformance project!

pyperformance/data-files/benchmarks/bm_asyncio_websockets/run_benchmark.py から始め、perf でプロファイルを再現して zlib の割合を確認します。リンクされている websockets の圧縮に関するガイダンスと、ベンチマークのドキュメントのエントリーポイントを確認します。maintainers が選択した解決策が実装され、ベンチマークの圧縮動作が明確に文書化されるか、測定対象のワークロードから除外されれば完了です。