[FR] Allow benchmarks to report their parameterization structurally to JSON
- Dominant language
- C++
- Stars
- 10.4k
- Forks
- 1.8k
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 8
Description
**Is your feature request related to a problem? Please describe.**
The `benchmark::RegisterBenchmark()` API supports adding benchmarks that are parameterized with arbitrary values, including custom names, lambdas, arbitrary objects, etc. The structural details of the parameterization is currently not available in the JSON output. The best the user can do is encode information in the benchmark name, which is a good idea anyway for clear console output. For the JSON output, however, it would be nice if a more data driven approach was available, such as key/value pairs. This way, analysis scripts need not parse the information out of the benchmark names.
I have personally come across a few cases where I have registered a benchmark matrix manually in code and had to come up with a benchmark name parsing approach when analyzing the data. This is an uncommon edge case, though, so I agree that any solution here should be simple.
This is similar to #838, but for the parameters that the benchmark library can't know about, doesn't know how to represent in JSON, etc.
**Describe the solution you'd like**
I am not invested in any particular solution, but this is one idea:
> Question is, what's the end use case? I think in general it is bad to enshrine in API something that is going to be
> rather obscurely used, but rather instead some generalization should be provided that allows to solve bigger problem.
> There's already global `AddCustomContext()`. Perhaps having the same but with different scopes will be sufficient?
_Originally posted by @LebedevRI in https://github.com/google/benchmark/issues/838#issuecomment-1242788747_
So perhaps code could look like this (pseudocode):
```c++
for (param1 : param1_values) {
for (param2 : param2_values) {
name = "BM_Thing/" + param1.ToString() + "/" + param2.ToString();
auto* benchmark = benchmark::RegisterBenchmark(
name, [param1, param2](benchmark::State& state) {
BM_MyBenchmark(state, param1.value(), param2.value());
});
benchmark->AddCustomContext("param1", param1.ToString());
benchmark->AddCustomContext("param2", param2.ToString());
}
}
```
Some of the fancier `BENCHMARK_` macro wrappers around `benchmark::RegisterBenchmark()` could perhaps be adapted to add their stringified args using this new `benchmark->AddCustomContext()` API.
**Describe alternatives you've considered**
In the past I thought about URL encoding key/value pairs in the benchmark name. ;-)
**Additional context**
Eventually, with something like this in place perhaps some of the filtering logic in `compare.py` could optionally match benchmarks by these values, rather than requiring the use of a regex against the name.
Contributor guide
Assessment
This issue has not been assessed yet.