GoogleCloudPlatform / GoogleCloudPlatform/PerfKitBenchmarker
Concerns with assumptions and limitations of UnixBench
- Dominant language
- Python
- Stars
- 2k
- Forks
- 562
- Avg merge
- 4h 55m
- Merged PRs (30d)
- 69
Description
I have several concerns with the current usage of UnixBench. I know that most people seems to run UnixBench 'as is' with whatever defaults are set, but I believe this will not show a proper comparison between systems (i.e. OS versions, Compiler versions, libC versions, etc).
Here's why:
- **Doesn't scale beyond 16 CPUs; at least not by default**
- This issue has been [known for a while and a patch is available](https://code.google.com/p/byte-unixbench/issues/detail?id=4), for some reason when UnixBench was written, a [hard-coded limit (16)](https://code.google.com/p/byte-unixbench/source/browse/trunk/UnixBench/Run#109) was set for the Systems and Non-Index benchmarks suites. Obviously, nowadays we have much more CPUs than that on single machine.
- This means that running UnixBench on a instance-type with 32 CPUs will give scores approximately the same as as 16 CPUs instance-type, assuming they use the same processors, of course.
- **GCC compiler options**
- The UnixBench Makefile by default [assumes a Solaris OS](https://code.google.com/p/byte-unixbench/source/browse/trunk/UnixBench/Makefile#74), even when the flags for [Linux are commented out](https://code.google.com/p/byte-unixbench/source/browse/trunk/UnixBench/Makefile#68), although it assumes a Pentium ISA.
- Compiling with different flags it's known to have huge impacts on the performance of a program, you can tweaks things like unrolling loops, to inlining functions, but more concerning, it's been shown recently that [even linking order and environment variables](http://www-plan.cs.colorado.edu/klipto/mytkowicz-asplos09.pdf) can impact performance significantly.
- This is a little trickier to solve, but perhaps one way to mitigate this is to statically compiled binaries for specific architectures, this way it will always run the same number of instructions, and be linked the same way.
- **GCC versions!**
- Brendan Gregg wrote this great post showing how wildly UnixBench can vary by just changing GCC versions. I'm just going to [link his blog post here](http://www.brendangregg.com/blog/2014-05-02/compilers-love-messing-with-benchmarks.html), which explains it better and was the inspiration for this issue.
What is your take on these issues?
Contributor guide
Assessment
This issue has not been assessed yet.