CI runs benchmarks but doesn't gate on them — add a main-vs-head comparison
- Dominant language
- Java
- Stars
- 615
- Forks
- 82
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 7
Description
## What is your idea?
Since #44, CI runs the `Benchmarks#CL2*` tests and the JMH suite on every PR
and push to main. But the results don't gate anything:
- The assertions inside the CL2 benchmarks check match-count correctness, not
throughput — a change that halves events/sec still passes.
- The numbers are head-only. There is no baseline from `main` in the same run,
so detecting a regression means a human manually pulling the benchmark step
output from two runs and diffing per JDK.
- Cross-runner noise on hosted runners is roughly ±10-15%, so naive absolute
thresholds would be flaky.
Releases inherit whatever CI enforces on the released commit — which today
does not include performance.
Ideas for closing the gap, roughly in order of effort:
1. **Same-runner main-vs-head comparison on PRs**: a CI job that builds and
benchmarks both the merge-base and the head on the same runner (the repo
already has `scripts/perf-compare.sh` doing this pattern locally), then
fails or warns above a relative threshold. Same-host comparison sidesteps
most of the runner-to-runner noise.
2. **Report, don't fail, at first**: post the per-rule-type delta as a PR
comment / job summary so reviewers see it, and only promote to a hard gate
once the noise band is understood.
3. **JMH with retained baselines**: store JMH JSON results from main as an
artifact and compare against them, accepting that cross-run comparisons
need a wider threshold.
Option 1 + 2 (comparison job that publishes a delta summary, non-blocking
initially) seems like the best first step.
## Would you be willing to make the change?
Yes
## Additional context
#44 (made benchmarks run in CI), #26 / #105 (benchmark coverage gaps),
`scripts/perf-compare.sh`.
Contributor guide
Research direction
Start with scripts/perf-compare.sh and inspect how the existing CI runs the CL2 benchmarks and JMH suite. Design a same-runner merge-base-versus-head comparison that reports per-rule-type deltas without initially blocking the job, then verify the summary is published for pull requests and pushes as intended.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, shell
- Domain
- ci-cd, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100