rust-lang / rust-lang/rustc-perf

Provide better analysis assistance in the comparison page

Open
#909 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A-ui C-feature-request
Dominant language
Rust
Stars
733
Forks
189
Avg merge
10h 52m
Merged PRs (30d)
43

Description

We currently only offer minimal assistance in analysis on the comparison page:

  • Noisy benchmarks are marked as such (though confusingly, it doesn't matter if one benchmark/incr-cache-state is noisy, the entire benchmark gets marked as noisy)
  • Changes above +- 0.2% get marked in light green or red, and changes changes above +- 1.0% get marked in darker green or red.

Now that we have an automated understanding of benchmark noisiness, we can do a lot better. Here are some ideas:

  • Flatten the benchmark data so that all benchmark/incr-cache-state pairs are shown in the same table without expanding
  • By default, filter all "non-significant" changes. We need to decide what it means to be significant. We have various definitions.
  • Make the green/red highlighting depend also on if the benchmark is noisy or not.
  • Better expose how a particular benchmark is noisy (i.e., is the benchmark actually noisy or just highly variable)
  • Show statistics of how many benchmarks showed significant changes in each direction.
  • Show correlation between cache-state/profile and significant changes (i.e., if all the significant changes were in optimized builds, we should show this)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the comparison page and inspect the existing benchmark-noisiness analysis described here, focusing on the remaining noise-aware highlighting and noisy-cause visibility work. Done means the comparison explains noise status and uses it in highlighting; no file or test is named in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.