Inconsistent continous benchmarking
- Dominant language
- Rust
- Stars
- 1.5k
- Forks
- 132
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 39
Description
Our continuous benchmarking setup (https://hermit-os.github.io/hermit-bench/) is lacking. It is run on our local VM cluster, which leads to a high variance in measured run times. While reworking our stack with [Bencher](https://bencher.dev/docs/tutorial/self-hosted) might be interesting, even running bare-metal is not perfectly consistent.
Instead of running with KVM, we could try emulation instead. QEMU's TCG supports [instruction counting](https://www.qemu.org/docs/master/devel/tcg-icount.html). There is an example TCG plugin that counts the number of instructions ([docs](https://www.qemu.org/docs/master/about/emulation.html#instruction), [tests/tcg/plugins/insn.c](https://gitlab.com/qemu-project/qemu/-/blob/v11.0.3/tests/tcg/plugins/insn.c)). While this will run slower and will not provide wall-time measurements, these should be extremely consistent and show even small changes in instruction count.
A slower, but more accurate way to gauge performance would be to run Hermit on a CPU simulator such as [gem5](https://www.gem5.org).
An alternative would be to explore Valgrind support to instructions, L1 hits, LL hits, RAM hits, total read+write, and estimated cycles.
Hardware performance counters would also be interesting, but they are often vendor- and CPU-specific.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing the continuous benchmarking setup at hermit-bench, then compare QEMU TCG instruction counting with the referenced tests/tcg/plugins/insn.c. The issue also lists gem5, Valgrind, and hardware counters as alternatives, so a contributor must first determine which approach to pursue. Done requires a selected and implemented method producing consistent benchmark measurements.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- operating-systems, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100