hermit-os / hermit-os/kernel

Inconsistent continous benchmarking

Open
#2,590 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
1.5k
Forks
132
Avg merge
1d 14h
Merged PRs (30d)
39

Description

Our continuous benchmarking setup (https://hermit-os.github.io/hermit-bench/) is lacking. It is run on our local VM cluster, which leads to a high variance in measured run times. While reworking our stack with [Bencher](https://bencher.dev/docs/tutorial/self-hosted) might be interesting, even running bare-metal is not perfectly consistent.

Instead of running with KVM, we could try emulation instead. QEMU's TCG supports [instruction counting](https://www.qemu.org/docs/master/devel/tcg-icount.html). There is an example TCG plugin that counts the number of instructions ([docs](https://www.qemu.org/docs/master/about/emulation.html#instruction), [tests/tcg/plugins/insn.c](https://gitlab.com/qemu-project/qemu/-/blob/v11.0.3/tests/tcg/plugins/insn.c)). While this will run slower and will not provide wall-time measurements, these should be extremely consistent and show even small changes in instruction count.

A slower, but more accurate way to gauge performance would be to run Hermit on a CPU simulator such as [gem5](https://www.gem5.org).

An alternative would be to explore Valgrind support to instructions, L1 hits, LL hits, RAM hits, total read+write, and estimated cycles.

Hardware performance counters would also be interesting, but they are often vendor- and CPU-specific.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the continuous benchmarking setup at hermit-bench, then compare QEMU TCG instruction counting with the referenced tests/tcg/plugins/insn.c. The issue also lists gem5, Valgrind, and hardware counters as alternatives, so a contributor must first determine which approach to pursue. Done requires a selected and implemented method producing consistent benchmark measurements.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
operating-systems, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.