catchorg / catchorg/Catch2

`--benchmark-samples` does not seem to work as expected

Open
#2,813 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
21.5k
Forks
3.5k
Avg merge
3d 16h
Merged PRs (30d)
2

Description

Hi all,

I have some functions I'm trying to benchmark, and some of them can only be called once (internal mechanisms prevent a user to call them twice). The benchmarked function **cannot** be modified for obvious reasons, and there is no way I can prevent them from checking they are called only once.

After few search, I found the `--benchmark-samples` option, which allows to set the number of times the code within `BENCHMARK` macro is called. Unfortunately it seems it does not act as intended. I even used all the arsenal of available options to control benchmarks as follows `--benchmark-samples 1 --benchmark-resamples 1 --benchmark-warmup-time 0 --benchmark-no-analysis`

Nothing changed.
What am I doing wrong? What should I do instead ?

Here is a sample code that produces the issue.

```C++
static uint8_t calls = 0;
static void function_callable_only_once() {
if (calls > 0) throw std::exception("Only one call is allowed.");
calls += 1;
}

BENCHMARK("generate_setup_commitments") {
std::cout << "********** THIS IS SPARTA *************" << std::endl;
REQUIRE_NOTHROW(function_callable_only_once());
};
```

The output is the following :
```bash
benchmark name samples iterations mean
-------------------------------------------------------------------------------
generate_setup_commitments ********** THIS IS SPARTA *************
1 1 ********** THIS IS SPARTA *************

stupid_test.cpp:25: FAILED:
REQUIRE_NOTHROW(function_callable_only_once())
due to unexpected exception with message:
Only one call is allowed.
```

Where you can see that the string `********** THIS IS SPARTA *************` is printed more than once.
Please help/advise.

Thanks.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.