google / google/benchmark

[FR] Consider adding extra info to the reducing variance doc relating to CPU frequency scaling

Open
#1,684 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
10.4k
Forks
1.8k
Avg merge
2d 4h
Merged PRs (30d)
8

Description

**Is your feature request related to a problem? Please describe.**

I'm trying to reduce variance in a benchmark, and doing all the things listed in the "reducing variance" doc, here [1].
This works fine on the box on my desk (where I have full control), but when I try and run it in the cloud then variance is back.
Why? AFAICT it's because the one "reducing variance" measure which does not appear to work with most cloud providers is pinning CPU frequency scaling.
And it doesn't seem to make a difference whether it's a regular cloud VM, or a cloud "bare metal" box.
I'm guessing this is because if you allow users to manipulate the CPU frequencies any way they like, then there is a risk of overheating the box?

**Describe the solution you'd like**

Add a section to the "reducing variance" doc [1], discussing the difficulties and possible solutions of implementing the measures locally versus in the cloud.

**Describe alternatives you've considered**

Have been Googling around a lot trying to find confirming or deny that "CPU frequency scaling" in some cloud provider somewhere works or doesn't work. Or that maybe local (non-cloud) benchmorks are the only way to go if CPU frequency scaling is needed?

**Additional context**

n/a

[1] https://github.com/google/benchmark/blob/main/docs/reducing_variance.md

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.