bytecodealliance / bytecodealliance/sightglass

Account for varying CPU frequency more robustly

Open
#138 17 comments 0 reactions 0 assignees View on GitHub
Dominant language
WebAssembly
Stars
88
Forks
37
Avg merge
5h 5m
Merged PRs (30d)
1

Description

Most modern CPUs scale their clock frequency according to demand, and this CPU frequency scaling is always a headache when running benchmarks. There are two main dimensions in which this variance could cause trouble:

- Varying frequency across time: if the CPU load of benchmarking causes the CPU to ramp up its frequency, then different benchmark runs could observe different results based on different CPU frequency.
- Varying frequency across space: if different CPU cores are running at different frequencies, then benchmark runs might intermittently experience very different performance if they are not pinned to specific cores.

I've been seeing some puzzling results lately and I suspect at least part of the trouble has to do with the above. I've set my CPU cores to the Linux kernel's `performance` governor, but even then, on my 12-core Ryzen CPU, I see clock speeds between 3.6GHz and 4.2GHz, likely due to best-effort frequency boost (which is regulated by thermal bounds and so unpredictable).

Note that measuring only cycles does not completely remove the effects of clock speed, because parts of performance are pinned to other clocks -- e.g., memory latency depends on the DDR clock, not the core clock, and L3 cache latency depends on the uncore clock.

The best ways I know to avoid noise from varying CPU performance are:

- Have longer benchmarks. Some of the benchmarks in this suite are only a few milliseconds long; this is not enough time to reach a steady state.
- Interleave benchmark runs appropriately. Right now, it looks like the top-level runner does a batch of runs with one engine, then a batch of runs with another. If the runs for different engines/configurations were interleaved at the innermost loop, then system effects that vary over time would at least impact all configurations roughly equally.
- Pin to a particular CPU core. For single-threaded benchmarks, this is probably the most robust way to have accurate A/B comparisons: if cores have slightly different clock frequencies, just pick one of them. Even better would be to do many runs and average across them all, but in high-core-count systems, removing this noise would take a lot of runs (hundreds of processes); with a few (5-10) process starts, it's entirely possible for the variance in mean core speed to be significant.
- Observe CPU governor settings when on a known platform (Linux: `/sys/devices/system/cpu/cpu*/cpufreq/scaling_governor` text file, will usually be `ondemand`, we want`performance`) and warn if scaling is turned on

Thoughts? Other ideas?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.