Allow calculating geometric mean of groups of benchmarks based on tags
未关闭
还没有人认领这个 Issue。
- 主要语言
- Python
- 星标
- 1k
- 派生
- 203
- 平均合并
- 1 小时 20 分钟
- 30 天内合并 PR
- 2
描述
[Moved from https://github.com/faster-cpython/ideas/discussions/395]
It's becoming obvious that:
- The pyperformance suite needs more benchmarks that are more similar to real-world workloads, and we should lean into optimizing for these and using these to report progress.
- Microbenchmarks of a particular feature are also useful and belong in the benchmark suite, but we shouldn't over-optimize for them or use them as a (misleading) indicator of overall progress.
It seems that one way to address this would be to lean into "tags" more in the pyperformance/pyperf ecosystem. pyperformance already allows for tags in each benchmark's pyproject.yaml.
I propose we:
- Output the tags for each benchmark in the benchmark results in the
metadatadictionary. pyperfcompare_towould then calculate the geometric mean for each subset of benchmarks for each tag found in the results, as well as "all" benchmarks (existing behavior). This could be behind a flag if backward compatibility matters.
Alternatives:
We could instead use the nested benchmark heirarchy, rather than tags. Personally, I think tags is easier to understand and more flexible (a benchmark could be associated with multiple tags).
贡献指南
这个仓库没有索引到贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
调研方向
首先跟踪 pyperformance's pyproject.yaml 中的基准测试标签如何表示在基准测试结果和元数据中,然后检查 pyperf 的 compare_to 流程。完成的标准是:标签已暴露在元数据中,并且已针对每个标签子集以及所有基准测试计算几何平均值,同时解决向后兼容行为。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- python
- 领域
- performance, tooling
- Issue 类型
- 功能
- 难度
- 5/5
- 预计耗时
- 一周以上
- 活跃度
- 停滞
- 描述清晰度
- 基本清楚
- 新手友好度
- 32/100