algorand / algorand/go-algorand
Continuous benchmarking: Catch small performance regressions
- 主要语言
- Go
- 星标
- 1.4k
- 派生
- 537
- 平均合并
- 1 天 6 小时
- 30 天内合并 PR
- 17
描述
## Problem
https://github.com/algorand/go-algorand/tree/8bd8ab638214430c314c51a7869e0c874163360f/.github/workflows#benchmarking-performance-tests provides a continuous benchmarking solution with _loose_ alerting thresholds. Since the Github-provided runner environment varies significantly between runs, the solution catches _large_ performance regressions.
While it's helpful to catch large performance regressions, we want to monitor _small_ performance regressions without false positives. _Small_ is up for discussion. For starters, let's aim to diagnose performance drops >= 5%.
## Solution
Unfortunately, the simplest solution won't work out-of-the-box without excessive false positives. Tightening `alert-threshold` without any other change generates too much noise to be effective.
Possibilities include:
1. Replace Github-provided runners with self-managed hosts.
2. Rather than rely on cached before performance, evaluate before + after at same time.
### Suggested path
Try (2) with these considerations:
* Define Github Action with https://github.com/knqyf263/cob + do _not_ require a passing status check to merge PRs. That way, a performance regression does _not_ block PR merge.
* Time box either (a) extension of cob or (b) creation of an analogous tooling. Consider:
* Support for > 1 comparison point. Imagine comparing performance with the last 3 release tags.
* Interplay with Github Actions caching. Ideally, it's possible to reuse caches to speed up tests.
* Analogous to https://github.com/benchmark-action/github-action-benchmark:
* Marshalling test output to a graphing solution
* Adding a PR comment as a warning rather than failing the status check.
### Commentary for (1)
* Still subject to host variation.
* Assuming benchmarks are CPU bound, perhaps we can reduce variation by:
* Use AWS metal instances.
* Tune CPU governor to reduce CPU speed such that all hosts run at same speed.
* We can continue using github-action-benchmark by managing hosts via https://github.com/actions-runner-controller/actions-runner-controller.
### Commentary for (2)
* Effectively eliminates host variation at the expense of hindering the tracking of changes over time.
* Example tooling: cob (https://github.com/knqyf263/cob) offers a before + after evaluation technique.
* Perhaps tracking change over time is still possible. For example, imagine tracking relative performance changes instead of absolute measurements.
### Related links
* https://docs.github.com/en/actions/learn-github-actions/usage-limits-billing-and-administration#usage-limits
## Dependencies
## Urgency
贡献指南
评估
这个 Issue 还没有评估数据。