BENCH: consistently test benchmarks (specifically argmax/argmin)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 32.8k
- Forks
- 12.8k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 197
Description
In PR #17989 changes to unrelated code changed argmax/argmin benchmarks significantly. We should be more consistent about benchmarking PRs so we have a baseline. Perhaps the benchmark itself is unstable, perhaps somehow this PR changed some compiler branching. It would be nice to set up daily/weekly/per PR/ benchmark runs and compare them before merging PRs.
For this particular problem, we should try different dtypes and benchmark ordering to see what is causing the change in argmax/argmin.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing PR #17989 and the existing argmax/argmin benchmarks. Compare runs across different dtypes and benchmark orderings to determine whether the benchmark is unstable or compiler branching changed. Done means identifying the source of the variation and defining a repeatable benchmark baseline or run process for future PR comparisons.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ci-cd, performance, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100