RustPython / RustPython/RustPython

Make RustPython benchmarks readable

Open
#5,742 3 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
22.4k
Forks
1.5k
Avg merge
16h 33m
Merged PRs (30d)
165

Description

TL;DR: benchmarks are poorly readable and could be greatly improved. This is key element in convincing people of the soundness of RustPython so it should probably not be neglected IMHO.


The violin plots available here are not easily readable and their Y-axes labels are hardly readable at all because they got left-cut at some point. This is especially troublesome for the MICROBENCHMARKS section, for which it is impossible to tell RustPython from CPython.

This issue could be alleviated by doing the following:

  • Use a specific color for CPython and another one for RustPython (and keep this color pair consistent across all plots).
  • Always have CPython data on top and RustPython data on bottom (this is not consistent: in the EXECUTION tab, CPython is on top and RustPython on bottom, while in tab PARSE_TO_AST it is the other way around).
  • Only keep the name of the benchmark in the Y-axis labels, i.e. replace execution/mandelbrot.py/cpython by either Mandelbrot (and use a legend to indicate which color is which interpreter), or make a plot title saying Mandelbrot and use the Y-axis labels to tell whether it is CPython or RustPython.

In addition to these visual issues, some other improvements could be implemented:

  • Make the plots user-friendly using some interactive backend such as plotly.
  • Put hyperlinks to the benchmark script location / source-code, so that users can check what the benchmarks are actually doing.
  • In the same line of thought, add a small descriptive text about what the benchmark does / why it is relevant (for instance "benchmark X is particularly I/O intensive" or whatnot).
  • On top of the page, give the hash of the commit / version (possibly with release date to know at a glance if they're outdated or not) of both CPython and RustPython binaries that were used, whether they were recompiled with -o3 locally, as well as the machine specs (this would allow for meaningful comparison and reproducibility).

I think that benchmarks one of the key element that might convince anyone to switch from one interpreter to another (apart from functionalities / low-level bindings). Hence they should not be neglected.

If someone could point me to where these plots are generated, I'd be happy to help typesetting them / add further info (although I might need some technical support about why benchmark X is especially relevant or not).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the benchmarks page linked in the issue, especially the MICROBENCHMARKS, EXECUTION, and PARSE_TO_AST sections, and locate where these plots are generated. Compare the current labels, ordering, colors, and readability, then define a focused subset of improvements before changing broader metadata or interactivity. Done means the benchmark plots are consistently readable and distinguish CPython from RustPython.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-visualization, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.