coveragepy / coveragepy/coveragepy

"source" config combined with pytest-xdist leads to incorrect coverage

Open
#1,341 8 comments 13 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug subprocess
Dominant language
Python
Stars
3.4k
Forks
525
Avg merge
18h 27m
Merged PRs (30d)
6

Description

Describe the bug
Either source behaves weirdly in combination with xdist, or the documentation around
how source/include work could be improved.

When supplying certain source/include values in particular ways with xdist, you can
either get 0% or 100%.

In my real-life usecase, I actually see 16% coverage reported,
where the last xdist process reports some amount of coverage while everything
else reports 0. But perhaps due to the simpler test-setup i lay out below, it only
seems to result in 0s.

It seems, by description, similar to https://github.com/nedbat/coveragepy/issues/389

To Reproduce
How can we reproduce the problem? Please be specific. Don't link to a failing CI job. Answer the questions below:

  1. What version of Python are you using? 3.9.6
  2. What version of coverage.py shows the problem? 6.3.2
  3. What versions of what packages do you have installed? pytest-xdist==2.5.0
❯ tree
.
├── poetry.lock
├── pyproject.toml
├── src
│  └── foo
│     ├── __init__.py
│     └── meow.py
└── test.py
# pyproject.toml
[tool.poetry]
name = "foo"
version = "0.0.0"
description = ""
authors = []
packages = [
    { include = "foo", from = "src" },
]

[tool.coverage.report]
show_missing = true
skip_covered = true

[tool.coverage.run]
source = ["src"]
parallel = true
branch = true
# meow.py
def meow():
    print("meow")


def meow2():
    print("meow")


def meow3():
    print("meow")


meow()
# test.py
from foo.meow import meow, meow2, meow3


def test_meow():
    meow()


def test_meow2():
    meow2()


def test_meow3():
    meow3

If you run the following, you get the correct result (86% in this case)

coverage run -m py.test test.py && coverage combine && coverage report

If you run the following, you get 0%

coverage run -m py.test -n 3 test.py && coverage combine && coverage report

Perhaps notable other options I tried:

source = ["src/"]  # 0%
source = ["foo"]  # 0%
source = ["src/foo"]  # 0%
source = ["src/foo"]  # 0%
include = ["src"]  # 0%
include = ["src/*"]  # 86% hurray!

So tl;dr the behavior of source seems to be different with/without xdist, perhaps
in combination given the fact that the package is nested within a src/ directory
instead of foo/ being implicitly on the path.

*Expected behavior
I expect the behavior of the source option to react the same regardless of
use of the -n flag from pytest-xdist.

Alternatively, if this is difficult to work around, just that perhaps the docs
be made more clear that one needs to use include with a pattern.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Recreate the issue with the example pyproject.toml and test.py, comparing coverage run -m py.test test.py with coverage run -m py.test -n 3 test.py. First verify how the source and include values affect coverage combine and report in each case. Done means xdist produces the same coverage result as the non-xdist run, or the documentation clearly explains the required configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.