asyncio_websockets tests the implementation of `zlib`, not of Python

I found out that the asyncio_websockets benchmark spends ~87% of runtime in zlib (i.e. in the shared library libz.so), or whatever compression library is the default on the system-under-test.

In other words, asyncio_websockets tests the implementation of compression/decompression algorithms rather than anything to do with the Python interpreter or the websockets Python module. Websockets indeed enables compression by default: https://websockets.readthedocs.io/en/stable/topics/compression.html, excerpt from that official documentation:

connect() and serve() enable compression by default because the reduction in network bandwidth is usually worth the additional memory and CPU cost.

Problem

I believe this benchmark may not be measuring what's intended in its current form. For example, a replacement of zlib with zlib-ng or zlib-rs (which are newer drop-in replacement of zlib) may significantly affect the performance score of this benchmark, even though nothing changed in Python and/or websockets implementations. It is hard to root cause such performance modification, without knowing this detail about the asyncio_websockets benchmark.

It is also used in e.g. Phoronix testing, which may lead to unexpected conclusions for readers who aren't aware of this detail. Example: https://www.phoronix.com/review/cachyos-ubuntu-2510-f43/5.

Solutions

I see the following solutions:

1. Clearly document this behavior in https://pyperformance.readthedocs.io/benchmarks.html (in fact, there is no mention of websockets benchmark at all).
2. Remove this benchmark, since the workload is dominated by native zlib rather than Python or websockets logic.
3. Modify this benchmark to disable compression, as described here: https://websockets.readthedocs.io/en/stable/topics/compression.html#configuring-compression

I'd lean towards option 3 (disabling compression) as it preserves the benchmark's intent while removing the zlib dependency from results. I'm happy to submit a PR if the maintainers agree.

Reproducing

I ran it with a Amazon Linux 2023 docker container (OS distro similar to Fedora):

docker run --rm -it amazonlinux:2023 /bin/bash # -->

dnf install -y pip perf dnf-utils
dnf debuginfo-install zlib python3.9
python3 -m pip install pyperformance
python3 -m pip install websockets==11.0.3 pyperf==2.6.3

perf record -g --call-graph dwarf -- \
  python3 -u /usr/local/lib/python3.9/site-packages/pyperformance/data-files/benchmarks/bm_asyncio_websockets/run_benchmark.py

After running the benchmark, we can examine the resulting perf.data file:

$ perf report --hierarchy
...
-   99.93%        python3
   -   87.22%        libz.so.1.2.11
      +   40.11%        [.] inflate_fast
      +   36.28%        [.] deflate_slow
      ...
   +    5.20%        libpython3.9.so.1.0
   +    1.60%        libc.so.6
   ...

P.S. Thank you for maintaining the pyperformance project!

Beginne mit pyperformance/data-files/benchmarks/bm_asyncio_websockets/run_benchmark.py und reproduziere das Profil mit perf, um den Anteil von zlib zu bestätigen. Prüfe die verlinkten Hinweise zur websockets-Komprimierung und den Einstiegspunkt der Benchmark-Dokumentation. Als abgeschlossen gilt die Aufgabe, wenn die von den Maintainers gewählte Lösung implementiert ist und das Komprimierungsverhalten des Benchmarks eindeutig dokumentiert oder aus der gemessenen Arbeitslast ausgeschlossen ist.