catchorg / catchorg/Catch2

Test case fails but no assertion fails

Open
#1,656 4 comments 0 reactions 0 assignees View on GitHub
Possible bug
Dominant language
C++
Stars
21.5k
Forks
3.5k
Avg merge
3d 16h
Merged PRs (30d)
2

Description

**Describe the bug**

On the CI pipeline of one of my projects, I am experiencing a strange behaviour by Catch in a couple of tests: apparently, all ``REQUIRE()``s in all my ``TEST_CASE()``s pass, but one of the ``TEST_CASE()``s is marked as failing. See an example output here:

https://ci.appveyor.com/project/bluescarni/piranha-tng/builds/25230324/job/b2ymego1yi5kk852#L989

In the example above, the tests are run through CTest but the output and the result are identical if I run the offending executable manually: there is no information about what failed, and the exit status of the process is ``0``.

This happens *only* in a very specific case: a Windows build using ``clang-cl`` as a compiler, Visual Studio 2015 (as a standard library) and ``/std:c++17``. In the CI pipeline I have quite a few other build setups (MSVC 2019, Xcode, various versions of GCC/clang on ubuntu) and none of these builds exhibit the issue above. Some examples:

https://ci.appveyor.com/project/bluescarni/piranha-tng/builds/25230324/job/8drl58gx6gs5a832
https://travis-ci.org/bluescarni/piranha-tng/builds/544841099?utm_source=github_status&utm_medium=notification
https://circleci.com/gh/bluescarni/piranha-tng/1084?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link

How could I debug what is going on?

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure with Catch2 under clang-cl, Visual Studio 2015, and /std:c++17, first running the offending executable directly and then through CTest. Compare the reported test result, output, and exit status with the working MSVC, Xcode, and GCC/Clang configurations. Done means identifying why the test is marked failed without an assertion failure and documenting or fixing that behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.