Report other benchmark results relative to a baseline
- Dominant language
- C++
- Stars
- 10.4k
- Forks
- 1.8k
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 8
Description
Imagine the following scenario: I've build new and fancy `vector` and want to benchmark it. I could now write the benchmarks and get some numbers, but I would have absolutely no idea how good these numbers are. For simplicity, let's assume my benchmark tests `push_back`, and it tests pushing an integer 1M times. My benchmark runs in 15ms. That's just an absolute number, it doesn't give me slightest idea how this fares against `std::vector` (for example).
Therefore I'd love to see an equivalent to `BENCHMARK`, called `BENCHMARK_RELATIVE`. I would recreate the same benchmark for `std::vector`, and use it as a baseline.
```
template
static void BM_PushBack(benchmark::State &s)
{
Vec v;
v.reserve(1'000'000);
for (auto _ : s) {
v.clear();
for (int i = 0; i < 1'000'000; ++i)
v.push_back(i);
}
}
using BM_PushBackStd = BM_PushBack >;
using BM_PushBackCustom = BM_PushBack >;
BENCHMARK(BM_PushBackStd);
BENCHMARK_RELATIVE(BM_PushBackStd, BM_PushBackCustom);
```
The resulting output would somewhat like this:
```
2018-08-22 10:46:25
Run on (8 X 4000 MHz CPU s)
CPU Caches:
L1 Data 32K (x4)
L1 Instruction 32K (x4)
L2 Unified 262K (x4)
L3 Unified 8388K (x1)
----------------------------------------------------------------
Benchmark Relative Time CPU Iterations
----------------------------------------------------------------
BM_PushBackStd 30 ms 30 ms 41
BM_PushBackCustom 196.23% 15 ms 15 ms 22
```
What relative means is up to discussion, in this case I've defined `100%` as equal in relative speed, everything below `100%` is slower than baseline, and everything above is faster. `196%` means it's 1.69 times or 96% faster as baseline.
And credit where credit is due, this idea originally comes from folly's `benchmark.h`: https://github.com/facebook/folly/blob/master/folly/docs/Benchmark.md
Contributor guide
Assessment
This issue has not been assessed yet.