facebookresearch / facebookresearch/ProgramBench
pytest-rerunfailures 16.6.1 duplicates JUnit records and inflates scores
- Dominant language
- Python
- Stars
- 924
- Forks
- 67
- PR merge metrics
- No merged PRs in 30d
Description
### Problem
The evaluator installs `pytest-rerunfailures` without a version constraint. Version 16.6.1 changed its JUnit reporting behavior so that every rerun attempt produces a `` record.
For a test that still fails after two reruns, the XML contains two empty records followed by one failure record for the same `classname` and `name`:
```xml
AssertionError
```
`parse_test_results` currently processes each record independently and treats a record without a result child as passed. The single failing logical test is therefore scored as two passes and one failure, producing a score of 2/3 instead of 0/1.
### Reproduction
1. Install `pytest-rerunfailures==16.6.1`.
2. Run an always-failing test with `pytest --reruns=2 --junitxml=results.xml`.
3. Pass the resulting XML to `parse_test_results`.
With 16.6, the XML contains only the final failure record. With 16.6.1, it contains the three records shown above.
### Expected behavior
Each logical test should contribute exactly once to the score, using the final attempt's status. Retry-selection metrics should also count logical tests rather than raw JUnit records. The installed rerun plugin version should be reproducible.
A fix is available in #63.
Contributor guide
Assessment
This issue has not been assessed yet.