rust-lang / rust-lang/rustc-perf

Create initial version of a "dashboard" that shows long-term performance trends

Open
#192 16 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A-ui C-feature-request
Dominant language
Rust
Stars
732
Forks
189
Avg merge
10h 52m
Merged PRs (30d)
43

Description

This should implement https://github.com/rust-lang/rust/issues/48750, but for starters just containing the FROM-SCRATCH and SMALL-CHANGE usage scenarios.

I imagine this to be a separate page on perf.rlo, showing just two graphs, one for each usage scenario.

Each graph should contain a data point for each stable release (starting a few releases back, maybe with 1.17), the current beta, and the current nightly. The score should be calculated as the geometric mean of all build times for the given usage scenario, as listed in https://github.com/rust-lang/rust/issues/48750.

The main tasks to accomplish this are:

  • Add the missing benchmarks
    • ripgrep
    • webrender
    • cargo
    • winapi
    • stm32f103xx
    • encoding-rs (cargo test --lib --no-run)
    • clap-rs (cargo test --no-run)
    • regex (cargo test --lib --no-run)
    • syn (cargo test --no-run)
    • futures (cargo test --test=all --no-run)
  • Allow for running benchmarks in a Rust version specific way (so we don't crash when trying to specify -Cincremental on versions that don't support it yet)
  • Setup the new page (e.g. under perf.rlo/dashboard.html)

That page should make sure that all benchmark results for computing a given aggregate result are available, otherwise the result would be misleadingly low.

The new benchmarks should also be displayed on the regular perf.rlo page. Some open questions:

  • Do we need to start looking into running tests in parallel on multiple machines (because they are becoming more and more)?
  • What to do with the cargo test benchmarks? Probably just treat them as separate benchmarks, that is, they would add another row instead of another column: encoding-rs and encoding-rs (test), for example, would not be tied together in any way (although it would be nice if they could keep sharing the same source directory)

@Mark-Simulacrum, @nikomatsakis, thoughts?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the benchmark checklist in the issue and the existing perf.rlo page; review how the FROM-SCRATCH and SMALL-CHANGE results are currently produced. Implement the missing benchmarks and Rust-version-specific execution, then add perf.rlo/dashboard.html with release, beta, and nightly graphs whose geometric means include only complete inputs. Verify that the new benchmarks also appear on the regular perf.rlo page.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-visualization, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.