ChristianMurphy / ChristianMurphy/pytest-crap
Bug: Function coverage in CRAP report counts physical lines, not executable coverage lines
- Dominant language
- Python
- Stars
- 3
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
It seems that pytest-crap reports function coverage that is much lower than what is reported by pytest-cov.
### Minimal reproduction
`crap.py`:
```python
def func() -> int:
"""
This is a function with a long docstring.
The docstring explains a lot of complicated things, a lot of ins, a lot of
outs, a lot of what-have-yous. The docstring itself does not need to be
tested, pytest-cov does not count it as tested, and pytest-cov not reporting
it as tested is not a sign of poor software quality.
"""
return 0
```
`test_crap.py`:
```python
from crap import func
def test_crap() -> None:
assert func() == 0
```
Run:
`$ pytest --cov=crap --crap`
### Actual result
pytest-cov correctly reports 100% coverage, but pytest-crap reports 20% coverage, giving a CRAP score of 1.51 instead of the expected 1.
### Expected result
func should report 100% coverage, because all executable statements in the function are covered.
### Suspected cause
It looks like pytest-crap calculates function coverage as:
executed physical lines in function span / total physical lines in function span
In this example, the function spans lines 1-10, but only the def line and return line are executable/recorded by coverage.py. The docstring lines are counted in the denominator even though they are not uncovered executable statements.
So the plugin reports:
2 / 10 = 20%
This makes the coverage metric depend on docstrings, comments, blank lines, and formatting rather than actual untested executable code. Since the CRAP formula uses this coverage value, the CRAP score is also distorted.
A better behavior would be to compute per-function coverage using coverage.py's executable/missing statement data within the function's line range, rather than all physical source lines. This would obviously change the metric reported by pytest-crap, so it could be considered a breaking change.
Contributor guide
Assessment
This issue has not been assessed yet.