CodSpeed reporting wrong numbers

The problem

I've seen quite a few times recently that benchmark measures CodSpeed gives on PRs are erroneous.

This is a problem when doing perf work, as you can't tell if what you're doing is good or not.

e.g. PR https://github.com/oxc-project/oxc/pull/4214 showed initially a giving 0 speed-up, but then benchmarks re-ran after the PR below it in the stack was merged, and suddenly it shows 6% perf improvement. https://codspeed.io/oxc-project/oxc/branches/07-12-perf_semantic_reduce_lookups

That's wrong. The PR gives 0 perf improvement.

Reason was that in the last run, CodSpeed did the comparison to 2 commits back (3016f03), rather than 1 back. So 6% result shown included the perf boost of https://github.com/oxc-project/oxc/pull/4213 which is the commit that preceded it.

Why?

I am not sure why this has started happening recently. Could be:

1. Changes at CodSpeed's end.
2. Caused by our switch to using Graphite merge queue.

Solutions

1. Raise with CodSpeed.
2. If they can't fix, investigate if we can handle it somehow at our end.

Because we intercept and store bench results and upload them to CodSpeed our end, we could potentially get our Github action to check that benchmarks for previous commit have completed and been uploaded to CodSpeed already, before submitting results for current commit. If not, wait until they are.

Start by comparing the CodSpeed results for PRs 4214 and 4213, then inspect the GitHub Actions workflow that stores and uploads benchmark results. Check why the comparison used commit 3016f03 instead of the immediately preceding commit, and confirm that a fix selects the correct baseline or waits for the prior benchmark upload.