benchmark-action / benchmark-action/github-action-benchmark

Running go benchmarks: Use `-count` or not?

Open
#229 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
TypeScript
Stars
1.3k
Forks
184
PR merge metrics
No merged PRs in 30d

Description

We're setting up this action for our performance-critical go applications, and notice some variance in the results. In one situation it got so bad (or good) that PR branches now report performance regressions because the master branch by chance recorded a really good value.

I was wondering whether we can "smooth" that a bit by using `-count` when running the benchmarks, but am not sure how/if this action will handle that? If it "should just work", do you think it's a good idea?

Contributor guide

Open the contributing guide

Research direction

No files or tests are named. Start by reproducing a Go benchmark run with and without `-count` through the GitHub Action, then inspect how the action invokes and processes benchmark results. Done means the action's support for repeated benchmark counts is established, with behavior documented or implemented and covered by regression testing.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, go, typescript
Domain
ci-cd, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.