astral-sh / astral-sh/ruff

docstring code formatter: add benchmarks

Open
#8,909 0 comments 1 reaction 0 assignees View on GitHub
docstring formatter performance
Dominant language
Rust
Stars
49.6k
Forks
2.4k
Avg merge
2d 1h
Merged PRs (30d)
445

Description

There is some concern over whether the new docstring code snippet formatting could slow down the overall formatting process. To ameliorate this and understand its performance characteristics, we should try to write some micro-benchmarks that will help us track performance over time. @MichaReiser in particular had some suggestions here: https://github.com/astral-sh/ruff/issues/8857#issuecomment-1830959726

I think we should try to include some "real world" examples of code snippets, but it might also be useful to try to include some "tortured" examples too. That is, try to devise the absolute worst case for performance and see how it does. This will give us an idea of what perf might look like at the extremes. For example, this might help us characterize the costs of us checking whether the reformatted code is valid or not. See #8857.

At present, I have done [some very loose ad hoc benchmarking](https://github.com/astral-sh/ruff/issues/8857#issuecomment-1830018211) to get a general sense of things:

> For an ad hoc benchmark, if I run the formatter with and without `docstring-code` enabled, then runtime is about the same. I ran it a few times and couldn't notice a difference. (Not that this is a good substitute for a real benchmark, but perhaps suggestive that docstring code formatting doesn't have a huge impact on perf.)

In particular, I did that on CPython and polars.

Contributor guide

Open the contributing guide

Research direction

Read the docstring-code formatting discussion in issue #8857 and review the existing formatter benchmarking approach before choosing cases. Add benchmarks covering real-world and deliberately worst-case Python snippets, including the CPython and polars examples, and make sure they expose the performance cost of validating reformatted code over time.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
performance, testing
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.