catchorg / catchorg/Catch2

Implement Property Based Testing/ Data Generators

Open
#850 52 comments 32 reactions 1 assignee Claimed by @philsquared View on GitHub
Feature Request
Dominant language
C++
Stars
21.5k
Forks
3.5k
Avg merge
3d 16h
Merged PRs (30d)
2

Description

This has been a long-standing promise/ point of discussion and a number of issues have been raised that amount to this, so I thought I'd write up the definite feature request to direct them all to.

There are two strands to this:

1. Data generators, or parametrised tests: i.e. you want to re-use the same core set of assertions with a range (possible a wide range) or different inputs - including the cross product of multiple possible inputs. In some cases you want to specifically control the inputs, in others generate it as an increasing range, or randomly. There's a variation where different *types* are used to parameterise. While strictly speaking that's a different feature I suspect it's all going to get tied up in them same solution, so I'll include that here too. We can always unbundle it later, if necessary.

2. Building on (1) is the idea of [Property-Based Testing.](http://hypothesis.works/articles/what-is-property-based-testing/) This is a more formally structured approach to working with generated ranges of inputs and also includes features such as *shrinking* (where failing inputs are automatically reduced to the simplest/ smallest possible failure). The tests themselves are for *properties* which are usually invariants that should always hold - although sometimes an alternate (e.g. simple or reference) implementation is compared against.

Support for generators was experimentally added in the very early days of Catch, but were never progressed. In fact quite a lot of work was done in support of them, and a few alternate implementations started. The main obstacle was interaction with the `SECTION` tracking infrastructure. That code was reworked with generators in mind a couple of years ago, and a new proof-of-concept generators implementation was successfully written against it. However by that time the goal was full property-based testing and the path there seemed tortuous in a C++03 constrained context, so the effort was deferred to such a time as we could at least rebase on C++11 (and ideally would leverage C++14 and beyond to allow for things like Ranges v3 to be supported in the generators part). C++11 rebasing was decided for Catch2 - with generators/ PBT being one of the main motivators. At time of this writing a proof-of-concept implementation of Catch2 exists which consists of a mostly rewritten core. That work has paused while the "Catch Classic" backlog is tamed but will resume again soon - with generators and PBT being one of the first big features to be worked on next.

I'll keep this issue open until the feature is ready so others can be closed in favour of it.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.