coveragepy / coveragepy/coveragepy

Expected performance of coveragepy?

Open
#1,862 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

support
Dominant language
Python
Stars
3.4k
Forks
525
Avg merge
18h 27m
Merged PRs (30d)
6

Description

I wonder what is the expected performance of coverage? In my case it looks like some tests run 2x slower than without coverage

I am using python 3.12 with standard unittest framework.

Also, I noticed there is kind of big overhead before anything starts running, when you enable coverage. My app starts in about 1s, but if I enable coverage it starts running tests after about ~13s.

Here is a small sample of tests that are run without coverage:

2024-09-26 08:48:12,662 1 INFO odoodb odoo.addons.base.models.ir_attachment: filestore gc 0 checked, 0 removed 
2024-09-26 08:48:12,662 1 INFO odoodb odoo.service.server: 8 post-tests in 3.11s, 1804 queries 
2024-09-26 08:48:12,662 1 INFO odoodb odoo.tests.stats: base_env_meta: 12 tests 0.12s 118 queries 
2024-09-26 08:48:12,662 1 INFO odoodb odoo.tests.stats: reports_project: 16 tests 3.11s 1804 queries 
2024-09-26 08:48:12,662 1 INFO odoodb odoo.tests.stats: sale_importer_project: 9 tests 0.73s 286 queries 
2024-09-26 08:48:12,662 1 INFO odoodb odoo.tests.stats: versatil: 43 tests 0.36s 230 queries 
2024-09-26 08:48:12,662 1 INFO odoodb odoo.tests.result: 0 failed, 0 error(s) of 50 tests when loading database 'odoodb' 
2024-09-26 08:48:12,662 1 INFO odoodb odoo.service.server: Initiating shutdown 
2024-09-26 08:48:12,662 1 INFO odoodb odoo.service.server: Hit CTRL-C again or send a second signal to force the shutdown. 
2024-09-26 08:48:12,706 1 INFO odoodb odoo.sql_db: ConnectionPool(used=0/count=0/max=64): Closed 2 connections 

Here same tests with coverage:

2024-09-26 08:50:09,710 1 INFO odoodb odoo.service.server: 8 post-tests in 5.98s, 1804 queries 
2024-09-26 08:50:09,711 1 INFO odoodb odoo.tests.stats: base_env_meta: 12 tests 0.22s 118 queries 
2024-09-26 08:50:09,711 1 INFO odoodb odoo.tests.stats: reports_project: 16 tests 5.97s 1804 queries 
2024-09-26 08:50:09,711 1 INFO odoodb odoo.tests.stats: sale_importer_project: 9 tests 1.01s 286 queries 
2024-09-26 08:50:09,711 1 INFO odoodb odoo.tests.stats: versatil: 43 tests 0.42s 230 queries 
2024-09-26 08:50:09,711 1 INFO odoodb odoo.tests.result: 0 failed, 0 error(s) of 50 tests when loading database 'odoodb' 
2024-09-26 08:50:09,711 1 INFO odoodb odoo.service.server: Initiating shutdown 
2024-09-26 08:50:09,711 1 INFO odoodb odoo.service.server: Hit CTRL-C again or send a second signal to force the shutdown. 
2024-09-26 08:50:10,044 1 INFO odoodb odoo.sql_db: ConnectionPool(used=0/count=0/max=64): Closed 2 connections

All tests run slower especially ones with lots of queries. Note, odoo is tightly coupled with database engine (postgres), so there are lots of tests that depend on database. Not sure if its relevant here.

My .coveragerc:

[run]
include =
    /opt/odoo/projects/monodoo/*
    /opt/odoo/projects/custom/*

omit =
    **/tests/*
    **/migrations/*

[report]
# Regexes for lines to exclude from consideration
exclude_lines =
    # Have to re-enable the standard pragma
    pragma: no cover

    # Don't complain if non-runnable code isn't run:
    if __name__ == .__main__.:

I run coverage like this:

os.execvp(cmd, cmd_args)

Where cmd is coverage and cmd_args = ['coverage', 'run', '/opt/odoo/odoo/odoo-bin', '-d', 'odoodb', '--stop-after-init', '--test-enable', '-u', 'my-module']

P.S. I also run it via docker container.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the timings with Python 3.12, the shown .coveragerc, and the coverage run command against the reported unittest/Odoo sample. Compare startup and query-heavy test durations with and without coverage, then determine whether the observed overhead is expected or indicates a coveragepy performance issue; the issue provides no source file or test to modify.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
databases, performance, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.