catchorg / catchorg/Catch2

marking tests as xfail

Open
#2,704 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
C++
Stars
21.5k
Forks
3.5k
Avg merge
3d 16h
Merged PRs (30d)
2

Description

In pytest, `xfail(strict=false)` (default xfail) the tests are run, but the test_suite does not fail if the individual test fails. This is different from skipping tests or negating them in that in most cases it should pass, but due to architecture, floating point errors or other kind of noises, the tests can fail.

And related to that, having a parametrized/generator tests that we expect at least one test to pass, e.g. in the above situation we can add some random noise so that the tests eventually pass.

Use-case:
Tests that check if an error is thrown in an unstable regime.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.