On-demand generation of comparison report for current and previous release of dotnet/runtime
- Dominant language
- F#
- Stars
- 773
- Forks
- 301
- Avg merge
- 6d 9h
- Merged PRs (30d)
- 15
Description
## Description
To improve microbenchmark triaging efficiency, it would be beneficial to have a report (e.g., in a form of dashboard) comparing the previous .NET release results with the current dotnet/runtime main. E.g., comparing `dotnet/runtime:release/9.0` and `dotnet/runtime:main` measurements.
The report should list all new regressions/improvements based on comparison between the two sets of measurements. To reduce false positives caused by bimodal and noisy microbenchmark, an averaging of the last X measurements could be introduced to smooth the graphs.
Additionally, the report should allow to filter-out microbenchmarks which improved/regressed by less than X %.
## Implementation Idea
1. Filter related microbenchmark results
- filter out all measurements where `BuildBranch=ref/heads/main`
- filter out all measurements where `BuildBranch=ref/heads/release/9.0`
2. Average the results from past X days based on Configuration (e.g., Mono x64 AOT LLVM) and TestName (microbenchmark name).
3. For each Configuration and TestName, calculate the diff.
4. Filter-out measurements with diff less than specified threshold.
5. Present the report
- Individual regressions/improvements (Configuration + TestName) sorted by magnitude.
- Number of regressions/improvements per Configuration
---
cc: @kotlarmilos
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.