catchorg / catchorg/Catch2

Write test report with assumed failure prior to test case execution

Open
#2,445 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
21.5k
Forks
3.5k
Avg merge
3d 16h
Merged PRs (30d)
2

Description

**Description**
For the non-streaming, file based reporters (e.g. JUnit) a *fatal* crash of the test runner should still produce a meaningful report.

Meaningful in the sense, that any test case which had *started* execution should be marked as a fail even if Catch2 had encountered a fatal error, and every not started test should be marked as "not executed".

It is not possible to capture the execution time, or any test output of the crashing test case, but that's usually a non-issue.

The the current state, any (hard) crash results in no report file being written for the corresponding test suite at all. That even includes not writing any failure reports for detected test case failures prior to a hard crash.

The assumption that any crash can be "handled" by just intercepting the signal is wrong, in any scenario with heap corruptions etc. it is just plain dangerous to "handle" in any form. So hard crashes are actually still a common occurrence.

In technical terms:
Provide the option to write out incremental reports immediately prior to the start of each test case execution.

**Additional context**
In CI integration attempt this yields a blind spot with Catch2, as any test suite which doesn't manage to reach the final write-out will effectively never even register with any of the globing based JUnit aggregators.

A hard crash, or a test timeout which results in termination by external watchdog, is a common scenario.

The most important information is which test case of a test suite has caused a crash, be it as spurious as it only can be. Running test cases as individual processes is not a feasible default choice. Loosing the stdout and/or captured context is less of an issue, as a crash caused by a memory access violation or alike has to be analyzed with additional tool support anyway.

The additional IO overhead from replacing the result file over and over again is still significantly lower than reloading the test executable with a potentially huge dynamic dependency tree and cached state.

The same feature had previously proven valuable in a proprietary fork of GTest in bigger projects, so the behavior description above is well tested with regard to the interoperability in CI integrations.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.