benchmark-action / benchmark-action/github-action-benchmark
Fork opitmised for pytest-benchmark (upstream improvements?)
- Dominant language
- TypeScript
- Stars
- 1.3k
- Forks
- 184
- PR merge metrics
- No merged PRs in 30d
Description
Hey, thanks for the great package! There was a number of additional data aspects I wanted to capture/utilise from pytest-benchmark (in particular test grouping) and so I created a fork optimised for this: https://github.com/aiidateam/github-action-benchmark, and you can see an example output here: https://aiidateam.github.io/aiida-core/dev/bench/ubuntu-18.04/django/
Now that I've just about finalised my implementation, I thought it would be good to feedback, since I sure at least some of this can be upstreamed to here 😄
I've outlined the changes in the README (and also the commit history is pretty readable), but I'll copy it here:
1. More metadata is saved about the benchmark run, for later comparison:
- CPU data (cores, processors, speed) is specifically saved, per commit, in the `cpu` key (uses [systeminformation](https://www.npmjs.com/package/systeminformation) npm package).
- The python version is extracted from the pytest-benchmark data in an `extra` key
- A `metadata` action option is available to save additional data
- This data is all shown in the data point's tooltip
2. The group name is saved for each test (see [pytest-benchmark markers](https://pytest-benchmark.readthedocs.io/en/latest/usage.html#markers)), or `null` if no group is given.
In the web-page rendering, tests are then arranged by the group they are in, which can be given a sub-heading, description, etc, and also a group can be "consolidated" into a single chart (with handling of differing data points).
3. A new `render-json-path` allows for a JSON file to be copied, which is used to configure the rendered web-page, e.g. for certain test suites and groups:
```json
{
"suites": {
"name": {
"header": "Test Suite Title",
"description": "Description of test suite."
}
},
"groups": {
"group1": {
"header": "Group 1 Title",
"description": "Description of group 1.",
"single_chart": true,
"xAxis": "date",
"backgroundFill": false
}
}
}
```
4. Split original `index.html` into multiple HTML/JS files in the `src/assets` folder. This allows for easier testing and development of the output web-page.
This has additionally allowed for:
- Adding `npm run serve`, for local development of output web-page (using [light-server](https://www.npmjs.com/package/light-server))
- Adding `overwrite-assets` option, to specify whether any existing assets should be overridden during a commit to `gh-pages`.
5. Improve formatting of charts:
- Color cycling for consolidated charts
- For legend, extract common test name prefix as title
- Data point tooltips: rounding values to 5 significant figures and better formatting of dates etc.
- Addition of the `chartjs-plugin-zoom` and a "Reset Zoom" button.
6. Add `commit-msg-append` option , useful for adding e.g. `[ci skip]` to commit message, but not having it as part of the test suite key in the data JSON.
7. Removed capture of commit author/committer, since it can be obtained from the commit id/url, and just bloats the data JSON.
8. Renamed `max-items-in-chart` -> `max-data-items`, to better describe its function of truncating the saved data during a commit.
Contributor guide
Assessment
This issue has not been assessed yet.