dotnet / dotnet/performance

On-demand generation of comparison report for current and previous release of dotnet/runtime

Open
#4,844 0 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
F#
Stars
773
Forks
301
Avg merge
6d 9h
Merged PRs (30d)
15

Description

## Description
To improve microbenchmark triaging efficiency, it would be beneficial to have a report (e.g., in a form of dashboard) comparing the previous .NET release results with the current dotnet/runtime main. E.g., comparing `dotnet/runtime:release/9.0` and `dotnet/runtime:main` measurements.

The report should list all new regressions/improvements based on comparison between the two sets of measurements. To reduce false positives caused by bimodal and noisy microbenchmark, an averaging of the last X measurements could be introduced to smooth the graphs.

Additionally, the report should allow to filter-out microbenchmarks which improved/regressed by less than X %.

## Implementation Idea
1. Filter related microbenchmark results
- filter out all measurements where `BuildBranch=ref/heads/main`
- filter out all measurements where `BuildBranch=ref/heads/release/9.0`
2. Average the results from past X days based on Configuration (e.g., Mono x64 AOT LLVM) and TestName (microbenchmark name).
3. For each Configuration and TestName, calculate the diff.
4. Filter-out measurements with diff less than specified threshold.
5. Present the report
- Individual regressions/improvements (Configuration + TestName) sorted by magnitude.
- Number of regressions/improvements per Configuration

---

cc: @kotlarmilos

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.