bench.js: easy/fast way to answer the question 'did I regress?'
- Dominant language
- JavaScript
- Stars
- 539
- Forks
- 59
- PR merge metrics
- No merged PRs in 30d
Description
Currently the benchmarks take upwards of 15 minutes to run. It's really hard to determine, for a given change, whether a performance regression was introduced.
Possible solutions:
1. only run each benchmark 3 times, instead of 9. This might lead to less stable numbers, but if it doesn't then it's a good solution.
2. define a 'fast' subset of benchmarks that's reasonably representative. You've been tracking the benchmarks over time, so it shouldn't be too hard to pick the few that have reliably indicated overall performance (least-squares analysis perhaps? :))
3. introduce a single large 'complete' benchmark. Even if it took 20s to run each time, that would only be 9*20s = 3 minutes total.
It would also be nice to be able to easily compute baseline numbers, e.g. by 'node bench.js --did-i-regress' performing a git clone into a temporary directory and running the benchmarks at origin/master, then running with your changes.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.