coveragepy / coveragepy/coveragepy
"source" config combined with pytest-xdist leads to incorrect coverage
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 3.4k
- Forks
- 525
- Avg merge
- 18h 27m
- Merged PRs (30d)
- 6
Description
Describe the bug
Either source behaves weirdly in combination with xdist, or the documentation around
how source/include work could be improved.
When supplying certain source/include values in particular ways with xdist, you can
either get 0% or 100%.
In my real-life usecase, I actually see 16% coverage reported,
where the last xdist process reports some amount of coverage while everything
else reports 0. But perhaps due to the simpler test-setup i lay out below, it only
seems to result in 0s.
It seems, by description, similar to https://github.com/nedbat/coveragepy/issues/389
To Reproduce
How can we reproduce the problem? Please be specific. Don't link to a failing CI job. Answer the questions below:
- What version of Python are you using? 3.9.6
- What version of coverage.py shows the problem? 6.3.2
- What versions of what packages do you have installed? pytest-xdist==2.5.0
❯ tree
.
├── poetry.lock
├── pyproject.toml
├── src
│ └── foo
│ ├── __init__.py
│ └── meow.py
└── test.py
# pyproject.toml
[tool.poetry]
name = "foo"
version = "0.0.0"
description = ""
authors = []
packages = [
{ include = "foo", from = "src" },
]
[tool.coverage.report]
show_missing = true
skip_covered = true
[tool.coverage.run]
source = ["src"]
parallel = true
branch = true
# meow.py
def meow():
print("meow")
def meow2():
print("meow")
def meow3():
print("meow")
meow()
# test.py
from foo.meow import meow, meow2, meow3
def test_meow():
meow()
def test_meow2():
meow2()
def test_meow3():
meow3
If you run the following, you get the correct result (86% in this case)
coverage run -m py.test test.py && coverage combine && coverage report
If you run the following, you get 0%
coverage run -m py.test -n 3 test.py && coverage combine && coverage report
Perhaps notable other options I tried:
source = ["src/"] # 0%
source = ["foo"] # 0%
source = ["src/foo"] # 0%
source = ["src/foo"] # 0%
include = ["src"] # 0%
include = ["src/*"] # 86% hurray!
So tl;dr the behavior of source seems to be different with/without xdist, perhaps
in combination given the fact that the package is nested within a src/ directory
instead of foo/ being implicitly on the path.
*Expected behavior
I expect the behavior of the source option to react the same regardless of
use of the -n flag from pytest-xdist.
Alternatively, if this is difficult to work around, just that perhaps the docs
be made more clear that one needs to use include with a pattern.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Recreate the issue with the example pyproject.toml and test.py, comparing coverage run -m py.test test.py with coverage run -m py.test -n 3 test.py. First verify how the source and include values affect coverage combine and report in each case. Done means xdist produces the same coverage result as the non-xdist run, or the documentation clearly explains the required configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100