Failure information for prior failing test executions is omitted from XML reports
- Dominant language
- Python
- Stars
- 398
- Forks
- 57
- PR merge metrics
- No merged PRs in 30d
Description
Given a flaky test such as:
```
def test_random():
assert bool(random.getrandbits(1))
```
The output of the JUnit XML contains no `` information when a test case fails and then subsequently passes.
It looks like this:
```
```
Other test frameworks and runners typically include this failure information like this so that automated tools can process flaky test information like this in a more structured format than plain-text logs.
```
def test_random():
> assert bool(random.getrandbits(1))
E assert False
E + where False = bool(0)
E + where 0 = <built-in method getrandbits of Random object at 0x7f93c1078610>(1)
E + where <built-in method getrandbits of Random object at 0x7f93c1078610> = random.getrandbits
test_sample.py:9: AssertionError
```
In order to avoid parsers mistaking the `` for deterministic, non-flaky failure, some test runners use ``. Even better, some test frameworks go so far as to mark the `` as `flaky="true"`. However, I'm not sure this library has that level of control to achieve these things, but if it does all the better for devs trying to fix flaky tests.
Contributor guide
Assessment
This issue has not been assessed yet.