beyond-all-reason / beyond-all-reason/RecoilEngine

Proposal: Engine Benchmarking via Github Actions + Cloud VMs

Open
#2,948 8 comments 0 reactions 0 assignees View on GitHub
area: devops/gitops area: tools
Dominant language
C++
Stars
679
Forks
290
Avg merge
3d 2h
Merged PRs (30d)
40

Description

## Proposal
Running benchmarks takes a while to do comprehensively on an optimization related PR. I could spent the time doing other things. Why not run benchmarks on cloud VMs? And then hook it into github actions so engine developers can trigger it on their optimization specific PRs?

I put together a benchmark runner for my own use and now I'm wondering if we should add it to the engine repo.

## Discussion

Some open questions:
* would you use this?
* how much do we care about proving/quantifying the improvement of a given optimization PR?
* **(ie. is this a real problem?)**
* if it were your money, how much would you pay for a suite of benchmarks to run against your PR?
* **(ie. how much is your time worth?)**
* is it reasonable to use BAR donation $ for this?
* **(ie. is this wasteful?)**
* when/how often should it be used?
* **(ie. how do we make it not wasteful?)**

## Context
I find myself spending more time running benchmarks and collecting the data than actually immersed in the engine changes I'm thinking through. There are many such changes, some simple (like changing our [matmul code](https://github.com/beyond-all-reason/RecoilEngine/pull/2940)) and some more complex (adding [dirty checks](https://github.com/beyond-all-reason/RecoilEngine/pull/2680) to animation code). The difference may only be a few % overall (which adds up! but is difficult to measure).

When I run benchmarks locally, I'm often running benchmarks multiple times in a row to try and get enough statistical significance to tell whether a change is actually faster or not. That ends up being a bunch of bespoke bash scripting a bunch of serial runs that i pull manually into google sheets and then try and generate tables/graphs off of.

So, a benchmark runner would have three main benefits:
* run MORE benchmarks than I otherwise would have time/patience for locally
* run these benchmarks FASTER than I would locally, because they can run in parallel off a cloud vendor
* collect and present the data for these benchmarks automatically right in the PR

## Costs
Variable. Running more or fewer scenarios, repeated iterations -- all affect pricing just based on wall time of spinning up instances.

I'm currently running on c2d-standard-16 instances, so 8 core/16 thread instances. Running on ephemeral spot VMs for cheapest possible hourly pricing.

Here's an example of running 10 vs 50 benchmarks each:
"quick" suite @ n=10:
Image

"full" suite @ n=50:
Image

now with visualizations too:
Image

Contributor guide

Open the contributing guide

Research direction

No target file or test is identified. Start by reviewing the existing benchmark runner, the GitHub Actions integration point, and the referenced optimization PRs (#2940 and #2680); the work is complete only when scope, cost, trigger policy, and automated PR benchmark reporting are agreed and implemented.

Written by the indexing model from the issue text.

Assessment

Tech stack
bash, cpp, github-actions
Domain
ci-cd, infrastructure, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.