python / python/mypy

Improve precision of mypy performance tracking

Abierto
#14,358 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

performance topic-developer
Lenguaje dominante
Python
Estrellas
20.6k
Forks
3.3k
Métricas de merge de PR
Métricas de PR pendientes

Descripción

We automatically track changes in mypy performance over time (#14187). Currently we can detect changes of at least 1.5% pretty reliably, but smaller changes are hard to detect. #14187 has some relevant discussion, such as this comment: https://github.com/python/mypy/issues/14187#issuecomment-1362312576

I'd estimate that a cumulative performance regression of around 15% in 2022 was due to changes that were below the 1.5% noise floor. Getting the detection threshold down to 0.5% or below could be quite helpful in finding and fixing regressions.

I looked at individual measurements, and it seems possible that measurements slowly fluctuate over time. I'm not entirely sure what might be causing this. Just increasing the number of iterations we measure probably won't help much, since different batches of runs will cluster around different averages.

Here are some things that could help:

  1. Interleave executions of current/previous builds and measure the delta. Instead of only collecting absolute performance values, interleave runs using the previous commit and the target commit and calculate the average delta. If performance gradually fluctuates over time, this should help.
  2. Further tweak the configuration of the runner machine for stability. See https://github.com/scala/scala-dev/issues/338 as suggested by @A5rocks.
  3. Collect samples over a long period of time (say, 1 sample every hour over 12 hours).
  4. Collect detailed profiling data for each commit and also highlight differences in the time spent in different parts of the mypy implementation. If a single function gets 2x slower, it could be easy to detect this way, even if the change in overall performance is well below the noise floor. This could be quite noisy due to renaming/splitting functions, etc.

I'm going to start by investigating if the idea 1 seems feasible.

Guía de contribución

Abrir la guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Línea de trabajo

Empieza revisando el trabajo de seguimiento del rendimiento en el issue #14187 y su discusión enlazada, y después investiga si intercalar las ejecuciones de las compilaciones anterior y objetivo puede reducir el ruido de las mediciones. No se nombran archivos ni tests; se considera terminado cuando se haya establecido un enfoque viable que detecte de forma fiable regresiones cercanas al 0,5 % o inferiores.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
python
Área
performance
Tipo de issue
Nueva funcionalidad
Dificultad
5/5
Tiempo estimado
Más de una semana
Estado de actividad
Estancado
Claridad
Necesita aclaración
Aptitud para principiantes
25/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.