trixi-framework / trixi-framework/Trixi.jl
Make it easier to re-run failed tests
- Dominant language
- Julia
- Stars
- 731
- Forks
- 166
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 25
Description
Right now, when a test (or multiple) fail during automated testing using GitHub Actions (or when run locally), you get an output like [this](https://github.com/trixi-framework/Trixi.jl/pull/632/checks?check_run_id=3408554816#step:6:11308):

To figure out what's wrong, you now have to look through the logs to find the error message (which is already cumbersome), but then - and this is imho even more annoying - you have to figure out which specific test actually failed such that you can re-run it locally during bug fixing.
What I thus propose is to implement a way that allows to easily re-run failed tests without hassle on my local machine. I do not know what the best way (or even a good and realistic way) of achieving this would look like, but I'm just putting out some ideas here:
* Include a short, auto-generated id with each test name that can then be used in a command such as `trixi_runtest("e4f87b")`
* Print out the "qualified name" of each failed tests after the test summary that can be used to directly run this test from an appropriate `trixi_runtest` function, e.g., `2D-Part3/Examples 2D/P4estMesh/elixir_advection_restart.jl`
* Print, after the summary, filename and line number of the `@test_trixi_include` statement that caused the test to fail
Contributor guide
Assessment
This issue has not been assessed yet.