Test case fails but no assertion fails
- Dominant language
- C++
- Stars
- 21.5k
- Forks
- 3.5k
- Avg merge
- 3d 16h
- Merged PRs (30d)
- 2
Description
**Describe the bug**
On the CI pipeline of one of my projects, I am experiencing a strange behaviour by Catch in a couple of tests: apparently, all ``REQUIRE()``s in all my ``TEST_CASE()``s pass, but one of the ``TEST_CASE()``s is marked as failing. See an example output here:
https://ci.appveyor.com/project/bluescarni/piranha-tng/builds/25230324/job/b2ymego1yi5kk852#L989
In the example above, the tests are run through CTest but the output and the result are identical if I run the offending executable manually: there is no information about what failed, and the exit status of the process is ``0``.
This happens *only* in a very specific case: a Windows build using ``clang-cl`` as a compiler, Visual Studio 2015 (as a standard library) and ``/std:c++17``. In the CI pipeline I have quite a few other build setups (MSVC 2019, Xcode, various versions of GCC/clang on ubuntu) and none of these builds exhibit the issue above. Some examples:
https://ci.appveyor.com/project/bluescarni/piranha-tng/builds/25230324/job/8drl58gx6gs5a832
https://travis-ci.org/bluescarni/piranha-tng/builds/544841099?utm_source=github_status&utm_medium=notification
https://circleci.com/gh/bluescarni/piranha-tng/1084?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link
How could I debug what is going on?
Contributor guide
Research direction
Reproduce the failure with Catch2 under clang-cl, Visual Studio 2015, and /std:c++17, first running the offending executable directly and then through CTest. Compare the reported test result, output, and exit status with the working MSVC, Xcode, and GCC/Clang configurations. Done means identifying why the test is marked failed without an assertion failure and documenting or fixing that behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100