aws / aws/event-ruler

CI runs benchmarks but doesn't gate on them — add a main-vs-head comparison

Open
#262 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
615
Forks
82
Avg merge
1d 22h
Merged PRs (30d)
7

Description

## What is your idea?

Since #44, CI runs the `Benchmarks#CL2*` tests and the JMH suite on every PR
and push to main. But the results don't gate anything:

- The assertions inside the CL2 benchmarks check match-count correctness, not
throughput — a change that halves events/sec still passes.
- The numbers are head-only. There is no baseline from `main` in the same run,
so detecting a regression means a human manually pulling the benchmark step
output from two runs and diffing per JDK.
- Cross-runner noise on hosted runners is roughly ±10-15%, so naive absolute
thresholds would be flaky.

Releases inherit whatever CI enforces on the released commit — which today
does not include performance.

Ideas for closing the gap, roughly in order of effort:

1. **Same-runner main-vs-head comparison on PRs**: a CI job that builds and
benchmarks both the merge-base and the head on the same runner (the repo
already has `scripts/perf-compare.sh` doing this pattern locally), then
fails or warns above a relative threshold. Same-host comparison sidesteps
most of the runner-to-runner noise.
2. **Report, don't fail, at first**: post the per-rule-type delta as a PR
comment / job summary so reviewers see it, and only promote to a hard gate
once the noise band is understood.
3. **JMH with retained baselines**: store JMH JSON results from main as an
artifact and compare against them, accepting that cross-run comparisons
need a wider threshold.

Option 1 + 2 (comparison job that publishes a delta summary, non-blocking
initially) seems like the best first step.

## Would you be willing to make the change?

Yes

## Additional context

#44 (made benchmarks run in CI), #26 / #105 (benchmark coverage gaps),
`scripts/perf-compare.sh`.

Contributor guide

Open the contributing guide

Research direction

Start with scripts/perf-compare.sh and inspect how the existing CI runs the CL2 benchmarks and JMH suite. Design a same-runner merge-base-versus-head comparison that reports per-rule-type deltas without initially blocking the job, then verify the summary is published for pull requests and pushes as intended.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, shell
Domain
ci-cd, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.