algorand / algorand/go-algorand

Continuous benchmarking: Catch small performance regressions

未关闭
#4,610 1 条评论 1 个 reaction 已指派 0 人 在 GitHub 查看
Team Scytale
主要语言
Go
星标
1.4k
派生
537
平均合并
1 天 6 小时
30 天内合并 PR
17

描述

## Problem

https://github.com/algorand/go-algorand/tree/8bd8ab638214430c314c51a7869e0c874163360f/.github/workflows#benchmarking-performance-tests provides a continuous benchmarking solution with _loose_ alerting thresholds. Since the Github-provided runner environment varies significantly between runs, the solution catches _large_ performance regressions.

While it's helpful to catch large performance regressions, we want to monitor _small_ performance regressions without false positives. _Small_ is up for discussion. For starters, let's aim to diagnose performance drops >= 5%.

## Solution

Unfortunately, the simplest solution won't work out-of-the-box without excessive false positives. Tightening `alert-threshold` without any other change generates too much noise to be effective.

Possibilities include:
1. Replace Github-provided runners with self-managed hosts.
2. Rather than rely on cached before performance, evaluate before + after at same time.

### Suggested path
Try (2) with these considerations:
* Define Github Action with https://github.com/knqyf263/cob + do _not_ require a passing status check to merge PRs. That way, a performance regression does _not_ block PR merge.
* Time box either (a) extension of cob or (b) creation of an analogous tooling. Consider:
* Support for > 1 comparison point. Imagine comparing performance with the last 3 release tags.
* Interplay with Github Actions caching. Ideally, it's possible to reuse caches to speed up tests.
* Analogous to https://github.com/benchmark-action/github-action-benchmark:
* Marshalling test output to a graphing solution
* Adding a PR comment as a warning rather than failing the status check.

### Commentary for (1)
* Still subject to host variation.
* Assuming benchmarks are CPU bound, perhaps we can reduce variation by:
* Use AWS metal instances.
* Tune CPU governor to reduce CPU speed such that all hosts run at same speed.
* We can continue using github-action-benchmark by managing hosts via https://github.com/actions-runner-controller/actions-runner-controller.

### Commentary for (2)
* Effectively eliminates host variation at the expense of hindering the tracking of changes over time.
* Example tooling: cob (https://github.com/knqyf263/cob) offers a before + after evaluation technique.
* Perhaps tracking change over time is still possible. For example, imagine tracking relative performance changes instead of absolute measurements.

### Related links
* https://docs.github.com/en/actions/learn-github-actions/usage-limits-billing-and-administration#usage-limits

## Dependencies

## Urgency

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。