trixi-framework / trixi-framework/Trixi.jl

Make it easier to re-run failed tests

Open
#820 2 comments 0 reactions 0 assignees View on GitHub
discussion enhancement testing
Dominant language
Julia
Stars
731
Forks
166
Avg merge
2d 18h
Merged PRs (30d)
25

Description

Right now, when a test (or multiple) fail during automated testing using GitHub Actions (or when run locally), you get an output like [this](https://github.com/trixi-framework/Trixi.jl/pull/632/checks?check_run_id=3408554816#step:6:11308):
![image](https://user-images.githubusercontent.com/3637659/130589110-87aef086-fb5d-4c9e-8fff-94447eec7482.png)

To figure out what's wrong, you now have to look through the logs to find the error message (which is already cumbersome), but then - and this is imho even more annoying - you have to figure out which specific test actually failed such that you can re-run it locally during bug fixing.

What I thus propose is to implement a way that allows to easily re-run failed tests without hassle on my local machine. I do not know what the best way (or even a good and realistic way) of achieving this would look like, but I'm just putting out some ideas here:
* Include a short, auto-generated id with each test name that can then be used in a command such as `trixi_runtest("e4f87b")`
* Print out the "qualified name" of each failed tests after the test summary that can be used to directly run this test from an appropriate `trixi_runtest` function, e.g., `2D-Part3/Examples 2D/P4estMesh/elixir_advection_restart.jl`
* Print, after the summary, filename and line number of the `@test_trixi_include` statement that caused the test to fail

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.