Allow calculating geometric mean of groups of benchmarks based on tags
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1k
- Forks
- 203
- Avg merge
- 1h 20m
- Merged PRs (30d)
- 2
Description
[Moved from https://github.com/faster-cpython/ideas/discussions/395]
It's becoming obvious that:
- The pyperformance suite needs more benchmarks that are more similar to real-world workloads, and we should lean into optimizing for these and using these to report progress.
- Microbenchmarks of a particular feature are also useful and belong in the benchmark suite, but we shouldn't over-optimize for them or use them as a (misleading) indicator of overall progress.
It seems that one way to address this would be to lean into "tags" more in the pyperformance/pyperf ecosystem. pyperformance already allows for tags in each benchmark's pyproject.yaml.
I propose we:
- Output the tags for each benchmark in the benchmark results in the
metadatadictionary. pyperfcompare_towould then calculate the geometric mean for each subset of benchmarks for each tag found in the results, as well as "all" benchmarks (existing behavior). This could be behind a flag if backward compatibility matters.
Alternatives:
We could instead use the nested benchmark heirarchy, rather than tags. Personally, I think tags is easier to understand and more flexible (a benchmark could be associated with multiple tags).
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing how benchmark tags from pyperformance's pyproject.yaml are represented in benchmark results and metadata, then inspect pyperf's compare_to flow. Done means tags are exposed in metadata and geometric means are calculated for each tag subset as well as all benchmarks, with backward-compatibility behavior resolved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- performance, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100