llvm / llvm/llvm-project

[libc++] PauseTiming and ResumeTiming dominate several sequence container benchmarks

Open
#216,055 0 comments 0 reactions 1 assignee Claimed by @ldionne View on GitHub
libc++ performance
Dominant language
LLVM
Stars
40.5k
Forks
18.7k
PR merge metrics
PR metrics pending

Description

Several benchmarks in `libcxx/test/benchmarks/containers/sequence/sequence_container_benchmarks.h` call `state.PauseTiming()`/`state.ResumeTiming()` once per single operation, in order to reset the container between iterations. A Pause/Resume pair costs over 700 ns (approximately measured on my machine), which can be orders of magnitude more than the operation being measured (especially for small ranges). That makes the benchmark extremely un-sensitive to changes in the actual code.

**Affected benchmarks include**:
- `insert(begin, input-iter, input-iter) (no realloc)`
- `insert(begin, input-iter, input-iter) (half filled)`
- `insert(begin, input-iter, input-iter) (near full)`
- `append_range() (into empty container)`
- `prepend_range() (into empty container)`

Those are using Pause/Resume without batching because the operations are O(n) and the reasoning was that the operation's timing would trump the Pause/Resume overhead. But it seems like that may not always be the case.

**Possible fixes**:
1. Drop the pause where the reset can be folded into the timed region. For example in `append_range()`/`prepend_range()`, construct the container inside the loop and let it destroy without any pause:
```cpp
for (auto _ : state) {
Container c;
c.append_range(in);
DoNotOptimizeData(c);
}
```
We already have precedent for that (see `// we assume the destructor doesn't dominate the benchmark` comments).

2. Switch to `KeepRunningBatch(BatchSize)`. Also consider raising the batch size.

3. Drop smaller size configurations, since they don't really measure more than the Pause/Resume overhead (for these benchmarks) right now.

Other fixes might be possible, and their effectiveness should be measured before committing to an approach.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.