Feature request: templated data test case
- Dominant language
- C++
- Stars
- 213
- Forks
- 149
- Avg merge
- 18h 17m
- Merged PRs (30d)
- 2
Description
I can generate tests from data using the data-driven API:
```cpp
constinit auto test_data = std::array{-1, 0, 1, 99};
BOOST_DATA_TEST_CASE(test_1, test_data)
{
// tests using `sample`
}
// 4 tests generated
```
I can also generate tests from types using the template test API:
```cpp
using test_types = std::tuple;
BOOST_AUTO_TEST_CASE_TEMPLATE(test_1, T, test_types)
{
// tests for `T`
}
// 3 tests generated
```
But what I often need to do is generate tests from *both* types *and* data:
| Type | Values |
| ------------- | -------------- |
| `int` | -1, 0, 1, 99 |
| `char` | 'a', 'B', 'z' |
| `std::string` | "foo"s, "bar"s |
The obvious way to do this:
```cpp
auto const test_data_int = std::array{-1, 0, 1, 99};
auto const test_data_char = std::array{'a', 'B', 'z'};
auto const test_data_string = std::array{"foo"s, "bar"s};
BOOST_DATA_TEST_CASE(test_1_with_int,
boost::unit_test::data::make(test_data_int))
{
// test 1 for `int` using `sample`
}
BOOST_DATA_TEST_CASE(test_1_with_char,
boost::unit_test::data::make(test_data_char))
{
// test 1 for `char` using `sample`
}
BOOST_DATA_TEST_CASE(test_1_with_string,
boost::unit_test::data::make(test_data_string))
{
// test 1 for `std::string` using `sample`
}
// 9 tests generated
```
But that’s obviously a lot of duplication. It gets even worse when you add another type, or another test. You can mitigate this somewhat by delegating all the tests to templated test functions (like: `template void test_1(T const& sample)`). But that still leaves a ton of boilerplate.
Another option is to do something like this:
```cpp
template
auto const test_data = std::array{};
template <>
auto const test_data = std::array{-1, 0, 1, 99};
template <>
auto const test_data = std::array{'a', 'B', 'z'};
template <>
auto const test_data = std::array{"foo"s, "bar"s};
using test_types = std::tuple;
BOOST_AUTO_TEST_CASE_TEMPLATE(test_1, T, test_types)
{
for (auto&& sample : test_data)
{
BOOST_TEST_CONTEXT("sample = " << sample)
{
// test 1 for `T` using `sample`
}
}
}
// 3 tests generated
```
But this only generates 3 tests (one for each type) and not 9 (one for each sample, for each type). There are also other issues that having individual test cases for each sample are supposed to solve, like that if anything throws unexpectedly in the inner test context, all the remaining tests for that type are aborted, and so on and so forth.
What I’d like to be able to do is:
```cpp
template
auto const test_data = std::array{};
template <>
auto const test_data = std::array{-1, 0, 1, 99};
template <>
auto const test_data = std::array{'a', 'B', 'z'};
template <>
auto const test_data = std::array{"foo"s, "bar"s};
using test_types = std::tuple;
BOOST_AUTO_TEST_CASE_???(test_1, test_types, T, test_data /*, sample_var */)
{
// test 1 for `T` (from the elements of tuple test_types)
// using `sample` (from the elements of dataset `test_data`)
}
// 9 tests generated
```
***
The interface above would work just fine for me. It also jibes with the rest of the Boost.Test API, and probably wouldn’t be too hard to implement. It’s probably all that’s needed.
But, while I was musing over the idea, self-medicating with various mental lubricants, I started to wonder if there might not be a more interesting way to do this. Perhaps it might be possible to define a templated test suite - a test suite generated for each of a set of types - within which a set of tests could be generated. For example:
```cpp
BOOST_TEST_?TEMPLATED_TEST_SUITE?_BEGIN(name, T)
BOOST_DATA_TEST_CASE_?(test_1, test_data)
{
// test 1 for `T`
// using `sample` (from the elements of dataset `test_data`)
}
BOOST_TEST_?TEMPLATED_TEST_SUITE?_END()
BOOST_TEST_?REGISTER_TEMPLATED_TEST_SUITES?(name, test_types);
// generates:
// name
// name::test_1(-1)
// name::test_1(0)
// name::test_1(1)
// name::test_1(99)
// name
// name::test_1('a')
// name::test_1('B')
// name::test_1('z')
// name
// name::test_1("foo"s)
// name::test_1("bar"s)
```
One interesting thing about this is that you could potentially specialize:
```cpp
BOOST_TEST_?TEMPLATED_TEST_SUITE?_BEGIN(name, T)
BOOST_DATA_TEST_CASE_?(test_1, test_data) {}
BOOST_DATA_TEST_CASE_?(test_2, test_data) {}
BOOST_TEST_?TEMPLATED_TEST_SUITE?_END()
BOOST_TEST_?TEMPLATED_TEST_SUITE?_SPECIALIZE_BEGIN(name, std::string)
BOOST_DATA_TEST_CASE_?(test_1, test_data) {}
// test_2 doesn't apply for std::string
BOOST_TEST_CASE_?(string_only_test) {}
BOOST_TEST_?TEMPLATED_TEST_SUITE?_SPECIALIZE_END()
```
But that might be more flexibility than is necessary, because you could also just do:
```cpp
BOOST_AUTO_TEST_CASE_???(test_1, test_types, T, test_data) {}
BOOST_AUTO_TEST_CASE_???(test_2, test_types_without_string, T, test_data) {}
BOOST_AUTO_TEST_CASE(string_only_test) {}
```
The more interesting thing about the idea of a templated test suite is that maybe you could *nest* them:
```cpp
using test_types2 = std::tuple;
BOOST_TEST_?TEMPLATED_TEST_SUITE?_BEGIN(name, T)
BOOST_TEST_?TEMPLATED_TEST_SUITE?_BEGIN(inner, U)
BOOST_DATA_TEST_CASE_?(test_1, test_data)
{
// test 1 for `T`
// using `sample` (from the elements of dataset `test_data`)
}
BOOST_TEST_?TEMPLATED_TEST_SUITE?_END()
BOOST_TEST_?REGISTER_TEMPLATED_TEST_SUITES?(inner, test_types2);
BOOST_TEST_?TEMPLATED_TEST_SUITE?_END()
BOOST_TEST_?REGISTER_TEMPLATED_TEST_SUITES?(name, test_types);
// generates:
// name
// name::inner
// name::inner::test_1(-1)
// name::inner::test_1(0)
// name::inner::test_1(1)
// name::inner::test_1(99)
// name::inner
// name::inner::test_1(-1)
// name::inner::test_1(0)
// name::inner::test_1(1)
// name::inner::test_1(99)
// name
// name::inner
// name::inner::test_1(-1)
// name::inner::test_1(0)
// name::inner::test_1(1)
// ... and so on
```
This could be handy for, for example, testing conversions to/from `T`s to `U`s.
However, all of this is far more complex than what I generally need, and if it’s worth consideration at all, it’s best left for some far future feature set.
***
So this is what I’d actually like to be able to do:
```cpp
// Assuming:
template
auto const test_dataset = /* */;
// ... specialized for each T of interest, as necessary
// And:
using tuple_like_of_test_types = std::tuple;
BOOST_AUTO_TEST_CASE_???(
test_name,
tuple_like_of_test_types,
T,
test_dataset
/*, sample_var (etc... same as for BOOST_DATA_TEST_CASE) */)
{
// test 1 for `T` (from the elements of tuple_like_of_test_types)
// using `sample` (from the elements of dataset `test_dataset`)
}
// The above generates a test case for each sample, for each type.
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.