There is no per-benchmark setup when arguments are used
- Dominant language
- C++
- Stars
- 10.4k
- Forks
- 1.8k
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 8
Description
One of the main reasons to set up to a test fixture is to factor out any expensive setup code for your test, so that it runs the minimum number of times.
However, I can't find the right place to put such setup code, assuming the setup depends on the parameters exposed by `State::range()`. Ideally, you'd call the setup once for each parameter combination. If you put it in the fixture constructor however, it will only be called once benchmark registered using that fixture so it's too coarse (and of course as a consequence `State::range()` isn't available there anyways).
If you put it in `Fixture::Setup` it will be called on _every_ benchmark method invocation, including the repeated invocations as the framework searches for the right number of iterations. This means the same data may be repeatedly generated, which also adds noise to the benchmark (e.g., because memory is repeatedly being allocated and de-allocated).
It would be ideal to have a setup method that is called "per-benchmark" (i.e., per line in the tabular output).
As a hacky workaround, one can write a `Setup()` method that checks the `range()` values against the values seen last time and only does the setup if they are different (if there was a `range()` method that returned the entire vector of args it would make that easier).
Contributor guide
Assessment
This issue has not been assessed yet.