docstring code formatter: add benchmarks
- Dominant language
- Rust
- Stars
- 49.6k
- Forks
- 2.4k
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 445
Description
There is some concern over whether the new docstring code snippet formatting could slow down the overall formatting process. To ameliorate this and understand its performance characteristics, we should try to write some micro-benchmarks that will help us track performance over time. @MichaReiser in particular had some suggestions here: https://github.com/astral-sh/ruff/issues/8857#issuecomment-1830959726
I think we should try to include some "real world" examples of code snippets, but it might also be useful to try to include some "tortured" examples too. That is, try to devise the absolute worst case for performance and see how it does. This will give us an idea of what perf might look like at the extremes. For example, this might help us characterize the costs of us checking whether the reformatted code is valid or not. See #8857.
At present, I have done [some very loose ad hoc benchmarking](https://github.com/astral-sh/ruff/issues/8857#issuecomment-1830018211) to get a general sense of things:
> For an ad hoc benchmark, if I run the formatter with and without `docstring-code` enabled, then runtime is about the same. I ran it a few times and couldn't notice a difference. (Not that this is a good substitute for a real benchmark, but perhaps suggestive that docstring code formatting doesn't have a huge impact on perf.)
In particular, I did that on CPython and polars.
Contributor guide
Research direction
Read the docstring-code formatting discussion in issue #8857 and review the existing formatter benchmarking approach before choosing cases. Add benchmarks covering real-world and deliberately worst-case Python snippets, including the CPython and polars examples, and make sure they expose the performance cost of validating reformatted code over time.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- performance, testing
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100