pytest-dev / pytest-dev/pytest
Performance regression on services heavy on logging starting from 6.0.0rc1
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 14.5k
- Forks
- 3.4k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 35
Description
Hello!
After updating the pytest version from 5.4.3 to 6.0.0rc1 in my service I noticed the tests performance regression.
I was very determined to find the root cause so I started to run tests on commits that have been added to 6.0.0rc1.
I have been testing it on python:3.9.18-slim-bookworm Docker image.
Below you can find the results:
v5.4.3 tests took: 0:03:34 - however on another try they took 4 minutes.
https://github.com/pytest-dev/pytest/commit/e48ac692dedc11e5c5c8722b426f5b575eeadea0 tests took: 0:04:05
https://github.com/pytest-dev/pytest/commit/075903dafa6ef6f62e2af67f336ee45f267feaa1 tests took: 0:05:03 - performance regression
https://github.com/pytest-dev/pytest/commit/43c465c9bf6d40bd579d62e63e883823368e1fde tests took: 0:06:33 - even bigger regression in comparison to previous commit
https://github.com/pytest-dev/pytest/commit/e27228a4e4afc941cac34672400016f59ee1184d tests took: 0:06:23
https://github.com/pytest-dev/pytest/commit/9310d67773433825ba568dba770e7202fe09c916 tests took: 0:08:57
https://github.com/pytest-dev/pytest/commit/d2d11a8bdcd382b0ac1e75af0a36cf826bcc3fa0 tests took: 0:07:34
https://github.com/pytest-dev/pytest/commit/5a6296a2d73d32010d4cddc4dd6cf6a289c01d9d tests took: 0:08:05
https://github.com/pytest-dev/pytest/commit/981b0969404bc4116659dae8fa6318cfbab2c942 tests took: 0:06:54
v7.4.4 tests took: 9:50 - not sure why that long but I'd like to start one step at the time and tackle the regression introduced in old versions first
Unfortunately my tests are not very consistent in terms of execution time and I didn't find an easy way to reproduce the issue that I could provide here. Yet they consistently take ~4mins on pytest==5.4.3 and more than 5 on 6.0.0rc1. I suspect this is somehow related to the fact that my service is logging a lot. I also noticed (but that I am not 100% sure) that on 6.0.0rc1 the tests are getting slower during test execution (by using --durations feature). Meaning that at the beginning they work fast but towards the end they are getting slower.
In the affected commit itself it is mentioned that the performance can be affected. I really hope that you know well the pytest internals and by reading above description you will be able to point the root cause and it will make sense.
Please let me know if you want me to run more tests or provide some logs.
I am sorry that I cannot provide simple way to reproduce this issue.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported timing difference between pytest 5.4.3 and 6.0.0rc1 in the Python 3.9.18-slim-bookworm Docker image, using the logging-heavy service and --durations output described in the issue. Compare the listed commits, especially 075903d and 43c465c, and identify the change associated with the regression; done means a reproducible cause or benchmarked fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python
- Domain
- performance, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100