ChristianMurphy / ChristianMurphy/pytest-crap

Bug: Function coverage in CRAP report counts physical lines, not executable coverage lines

Open
#3 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3
Forks
1
PR merge metrics
No merged PRs in 30d

Description

It seems that pytest-crap reports function coverage that is much lower than what is reported by pytest-cov.

### Minimal reproduction

`crap.py`:

```python
def func() -> int:
"""
This is a function with a long docstring.

The docstring explains a lot of complicated things, a lot of ins, a lot of
outs, a lot of what-have-yous. The docstring itself does not need to be
tested, pytest-cov does not count it as tested, and pytest-cov not reporting
it as tested is not a sign of poor software quality.
"""
return 0
```

`test_crap.py`:

```python
from crap import func

def test_crap() -> None:
assert func() == 0
```

Run:

`$ pytest --cov=crap --crap`

### Actual result

pytest-cov correctly reports 100% coverage, but pytest-crap reports 20% coverage, giving a CRAP score of 1.51 instead of the expected 1.

### Expected result

func should report 100% coverage, because all executable statements in the function are covered.

### Suspected cause

It looks like pytest-crap calculates function coverage as:

executed physical lines in function span / total physical lines in function span

In this example, the function spans lines 1-10, but only the def line and return line are executable/recorded by coverage.py. The docstring lines are counted in the denominator even though they are not uncovered executable statements.

So the plugin reports:

2 / 10 = 20%

This makes the coverage metric depend on docstrings, comments, blank lines, and formatting rather than actual untested executable code. Since the CRAP formula uses this coverage value, the CRAP score is also distorted.

A better behavior would be to compute per-function coverage using coverage.py's executable/missing statement data within the function's line range, rather than all physical source lines. This would obviously change the metric reported by pytest-crap, so it could be considered a breaking change.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.