Performance monitoring test
- Dominant language
- Python
- Stars
- 12.3k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 45
Description
Performance is an important strength of CuPy.
For better quality assurance, it is nice to have a way to "fixed-point observation" of performance.
By having a baseline set of benchmarks, we can:
* Correctly evaluate performance improvement for each pull-requests
* Detect unexpected performance regression
NumPy has a set of benchmark codes to run on [Airspeed Velocity](https://asv.readthedocs.io/en/latest/).
https://github.com/numpy/numpy/tree/master/benchmarks
I haven't tested them yet, but NumPy benchmark code seems so simple that we can borrow them and then do `s/np/cp/`. It may be a good start point.
https://github.com/numpy/numpy/tree/master/benchmarks/benchmarks
We may however want some additional metrics like:
* memory allocation performance
* comparison with NumPy
Contributor guide
Research direction
Start by reviewing the NumPy benchmark examples linked in benchmarks and benchmarks/benchmarks, along with the Airspeed Velocity documentation. Determine how CuPy benchmarks should establish fixed performance baselines and detect regressions. Done should include a documented benchmark setup that evaluates pull-request performance, with the proposed memory-allocation and NumPy-comparison metrics addressed or scoped.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- performance, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100