josephg / josephg/Chipmunk-js

bench.js: easy/fast way to answer the question 'did I regress?'

Open
#3 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
539
Forks
59
PR merge metrics
No merged PRs in 30d

Description

Currently the benchmarks take upwards of 15 minutes to run. It's really hard to determine, for a given change, whether a performance regression was introduced.

Possible solutions:
1. only run each benchmark 3 times, instead of 9. This might lead to less stable numbers, but if it doesn't then it's a good solution.
2. define a 'fast' subset of benchmarks that's reasonably representative. You've been tracking the benchmarks over time, so it shouldn't be too hard to pick the few that have reliably indicated overall performance (least-squares analysis perhaps? :))
3. introduce a single large 'complete' benchmark. Even if it took 20s to run each time, that would only be 9*20s = 3 minutes total.

It would also be nice to be able to easily compute baseline numbers, e.g. by 'node bench.js --did-i-regress' performing a git clone into a temporary directory and running the benchmarks at origin/master, then running with your changes.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.