google / google/benchmark

There is no per-benchmark setup when arguments are used

Open
#743 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
10.4k
Forks
1.8k
Avg merge
2d 4h
Merged PRs (30d)
8

Description

One of the main reasons to set up to a test fixture is to factor out any expensive setup code for your test, so that it runs the minimum number of times.

However, I can't find the right place to put such setup code, assuming the setup depends on the parameters exposed by `State::range()`. Ideally, you'd call the setup once for each parameter combination. If you put it in the fixture constructor however, it will only be called once benchmark registered using that fixture so it's too coarse (and of course as a consequence `State::range()` isn't available there anyways).

If you put it in `Fixture::Setup` it will be called on _every_ benchmark method invocation, including the repeated invocations as the framework searches for the right number of iterations. This means the same data may be repeatedly generated, which also adds noise to the benchmark (e.g., because memory is repeatedly being allocated and de-allocated).

It would be ideal to have a setup method that is called "per-benchmark" (i.e., per line in the tabular output).

As a hacky workaround, one can write a `Setup()` method that checks the `range()` values against the values seen last time and only does the setup if they are different (if there was a `range()` method that returned the entire vector of args it would make that easier).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.