Forest-vs-forest benchmarking (PRs)
- Dominant language
- Rust
- Stars
- 697
- Forks
- 200
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 65
Description
**Issue summary**
To catch regressions, we should (on request) run the `forest-tool benchmark` commands of the current PR against the latest released version. The benchmarks will be run on `fuzzy` and pasted in the PR that requested the run.
The output could look like this:
| Benchmark | PR | 0.13.1 | Change |
|---|---|---|---|
| `car-streaming` | 17s | 20s | 0.85 |
| `forest-encoding` | 45 | 46 | 0.98 |
| `graph-traversal` | 10s | 10s | 1.00 |
| `graph-traversal (mem)` | 3.8 GiB | 3.5 GiB | 1.09 |
Tasks:
- [ ] Create Rust script:
- [ ] Download a `mainnet` snapshot if one isn't available on disk.
- [ ] Fetch the previous release binaries of Forest.
- [ ] Compile the current PR in release mode.
- [ ] Run benchmark commands with `forest-tool` for both the PR and the latest release.
- [ ] Measure runtime and peak memory usage (measure directly, ignore output from `forest-tool`).
- [ ] Format data as a markdown table and (optionally) as JSON.
- [ ] If a benchmark command fails or doesn't exist, mark its results with a dash: `-`.
- [ ] The relative change is the PR measurement divided by the previous release's measurement.
- [ ] Trigger script on PR comments containing a keyword (i.e., `!bench`). See the linked workflow from `grpc_bench`.
**Other information and links**
Trigger workflow on comment: https://github.com/LesnyRumcajs/grpc_bench/blob/master/.github/workflows/issue_bench.yml
Contributor guide
Assessment
This issue has not been assessed yet.