DynamoRIO / DynamoRIO/dynamorio

TEST REPORTING BUG: Wrong issue number can be reported for failing tests that are ignored

Open
#6,607 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
3.2k
Forks
629
Avg merge
2d 15h
Merged PRs (30d)
31

Description

**Describe the bug**

Test runs can report the wrong issue number for ignored tests.

**To Reproduce**

See for example,
https://github.com/DynamoRIO/dynamorio/actions/runs/7731301531/job/21078628227#step:7:84492

```
====> FAILURE in debug-internal-64 <====
debug-internal-64: 902 tests passed, **** 6 tests failed, of which 2 were flaky, but ignoring 3 for i#2941: ****
(ignore: i#2941) code_api|client.attach_test
code_api|api.static_maps_mixup_novars_FLAKY
(ignore: i#2941) code_api,opt_memory|common.loglevel
(ignore: i#2941) code_api,opt_speed|common.decode-stress
code_api|tool.drcachesim.invariants
code_api|api.static_sideline_FLAKY
WARNING: maximum warning/error limit hit for release-external-64!
Manually verify whether it succeeded.
```

Note i#2941 is being reported as the issue for the attach_test failure.
We'll need to augment ignored test marking to let one specify the issue so that it makes it through to the test output.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.