Verilator simulations can be made an order of magnitude faster

Background Work

• Yes, I searched the mailing list
• Yes, I searched the documentation

Feature Description

For simple tests where we only need to run an assembly program, we can make the Verilator simulations more than order of magnitude faster, by removing some functionality from the test harness.

Motivating Example

On a an AMD EPYC 9554 3.75 GHz processors, single-thread Verilator simulation of a single-core RocketChip runs at about 10 kHz, but by stripping down the harness, we could make it run at 270~kHz (single-thread on EPYC), i.e., 27x faster.

Here is how I achieved a 27x speedup:

1. Remove TL monitors (WithoutTLMonitors as stated in the documentation.
2. Directly load the program as a hex file into the simulated RAM ($readmemhex), see here.
3. Exclusively handle +verbose in Verilog using a simplified all-Verilog harness

The last step has the most significant effect. It seems that Verilator really struggles with how verbose printing is handled through $c(...) PLI calls. Even when simulation is non-verbose there is a huge performance impact.

Start with the Verilator simulation harness and the referenced generators/chipyard/src/main/scala/example/simulation/HarnessExtMem.scala location. Read the Software-RTL-Simulation documentation, then compare the linked SimpleTestDriver.v approach with the existing harness. Done means supporting the simplified assembly-test path while preserving the requested simulation behavior and demonstrating the reported speedup.