google / google/benchmark

Run all tests without arguments

Open
#2,028 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
10.4k
Forks
1.8k
Avg merge
2d 4h
Merged PRs (30d)
8

Description

Hi! I find it confusing that the tests require arguments. I want to change it.

Here are my reasons:

1) Not all tests check for required arguments
https://github.com/google/benchmark/blob/7697796fe5a70edbc3394d0593ef82afeebb7ddf/test/spec_arg_test.cc#L63-L70 and so it's easy to run a test incorrectly.
2) There are GitHub issues mentioning that a test fails when the actual problem was that the correct arguments were not passed.
3) Arguments need to be duplicated between `test/CMakeLists.txt` and `test/BUILD` and they are actually not in sync now (`perf_counters_test` is run with different arguments).

My solution:

1) Create `test/default_arguments.h`:
```C++
// ...

void AddTestArguments(int &argc, char **&argv, std::initializer_list args={}) {
if (argc > 1) {
std::cout << "Warning: User is not expected to pass any command line arguments\n";
}
static std::vector new_argv;
new_argv.insert(new_argv.end(), argv, argv + argc);
new_argv.insert(new_argv.end(), args.begin(), args.end());
new_argv.push_back("--benchmark_min_time=0.01s");
argv = const_cast(new_argv.data());
argc = static_cast(new_argv.size());
}
```
2) Call it after each `main` function like this
```C++
int main(int argc, char* argv[]) {
AddTestArguments(argc, argv, {"--benchmark_counters_tabular=true"});
benchmark::MaybeReenterWithoutASLR(argc, argv);
RunOutputTests(argc, argv);
}
```
3) Cleanup the code

Sounds good?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.