python / python/mypy

Improve precision of mypy performance tracking

Open
#14,358 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

performance topic-developer
Dominant language
Python
Stars
20.6k
Forks
3.3k
PR merge metrics
PR metrics pending

Description

We automatically track changes in mypy performance over time (#14187). Currently we can detect changes of at least 1.5% pretty reliably, but smaller changes are hard to detect. #14187 has some relevant discussion, such as this comment: https://github.com/python/mypy/issues/14187#issuecomment-1362312576

I'd estimate that a cumulative performance regression of around 15% in 2022 was due to changes that were below the 1.5% noise floor. Getting the detection threshold down to 0.5% or below could be quite helpful in finding and fixing regressions.

I looked at individual measurements, and it seems possible that measurements slowly fluctuate over time. I'm not entirely sure what might be causing this. Just increasing the number of iterations we measure probably won't help much, since different batches of runs will cluster around different averages.

Here are some things that could help:

  1. Interleave executions of current/previous builds and measure the delta. Instead of only collecting absolute performance values, interleave runs using the previous commit and the target commit and calculate the average delta. If performance gradually fluctuates over time, this should help.
  2. Further tweak the configuration of the runner machine for stability. See https://github.com/scala/scala-dev/issues/338 as suggested by @A5rocks.
  3. Collect samples over a long period of time (say, 1 sample every hour over 12 hours).
  4. Collect detailed profiling data for each commit and also highlight differences in the time spent in different parts of the mypy implementation. If a single function gets 2x slower, it could be easy to detect this way, even if the change in overall performance is well below the noise floor. This could be quite noisy due to renaming/splitting functions, etc.

I'm going to start by investigating if the idea 1 seems feasible.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the performance-tracking work in issue #14187 and its linked discussion, then investigate whether interleaving executions of previous and target builds can reduce measurement noise. No files or tests are named; done means establishing a viable approach that reliably detects regressions near 0.5% or below.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.