facebookresearch / facebookresearch/ProgramBench

pytest-rerunfailures 16.6.1 duplicates JUnit records and inflates scores

Open
#64 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
924
Forks
67
PR merge metrics
No merged PRs in 30d

Description

### Problem

The evaluator installs `pytest-rerunfailures` without a version constraint. Version 16.6.1 changed its JUnit reporting behavior so that every rerun attempt produces a `` record.

For a test that still fails after two reruns, the XML contains two empty records followed by one failure record for the same `classname` and `name`:

```xml

AssertionError

```

`parse_test_results` currently processes each record independently and treats a record without a result child as passed. The single failing logical test is therefore scored as two passes and one failure, producing a score of 2/3 instead of 0/1.

### Reproduction

1. Install `pytest-rerunfailures==16.6.1`.
2. Run an always-failing test with `pytest --reruns=2 --junitxml=results.xml`.
3. Pass the resulting XML to `parse_test_results`.

With 16.6, the XML contains only the final failure record. With 16.6.1, it contains the three records shown above.

### Expected behavior

Each logical test should contribute exactly once to the score, using the final attempt's status. Retry-selection metrics should also count logical tests rather than raw JUnit records. The installed rerun plugin version should be reproducible.

A fix is available in #63.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.