benchmark-action / benchmark-action/github-action-benchmark

Fork opitmised for pytest-benchmark (upstream improvements?)

Open
#44 0 comments 2 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
1.3k
Forks
184
PR merge metrics
No merged PRs in 30d

Description

Hey, thanks for the great package! There was a number of additional data aspects I wanted to capture/utilise from pytest-benchmark (in particular test grouping) and so I created a fork optimised for this: https://github.com/aiidateam/github-action-benchmark, and you can see an example output here: https://aiidateam.github.io/aiida-core/dev/bench/ubuntu-18.04/django/

Now that I've just about finalised my implementation, I thought it would be good to feedback, since I sure at least some of this can be upstreamed to here 😄

I've outlined the changes in the README (and also the commit history is pretty readable), but I'll copy it here:

1. More metadata is saved about the benchmark run, for later comparison:
- CPU data (cores, processors, speed) is specifically saved, per commit, in the `cpu` key (uses [systeminformation](https://www.npmjs.com/package/systeminformation) npm package).
- The python version is extracted from the pytest-benchmark data in an `extra` key
- A `metadata` action option is available to save additional data
- This data is all shown in the data point's tooltip

2. The group name is saved for each test (see [pytest-benchmark markers](https://pytest-benchmark.readthedocs.io/en/latest/usage.html#markers)), or `null` if no group is given.
In the web-page rendering, tests are then arranged by the group they are in, which can be given a sub-heading, description, etc, and also a group can be "consolidated" into a single chart (with handling of differing data points).

3. A new `render-json-path` allows for a JSON file to be copied, which is used to configure the rendered web-page, e.g. for certain test suites and groups:

```json
{
"suites": {
"name": {
"header": "Test Suite Title",
"description": "Description of test suite."
}
},
"groups": {
"group1": {
"header": "Group 1 Title",
"description": "Description of group 1.",
"single_chart": true,
"xAxis": "date",
"backgroundFill": false
}
}
}
```

4. Split original `index.html` into multiple HTML/JS files in the `src/assets` folder. This allows for easier testing and development of the output web-page.
This has additionally allowed for:
- Adding `npm run serve`, for local development of output web-page (using [light-server](https://www.npmjs.com/package/light-server))
- Adding `overwrite-assets` option, to specify whether any existing assets should be overridden during a commit to `gh-pages`.

5. Improve formatting of charts:
- Color cycling for consolidated charts
- For legend, extract common test name prefix as title
- Data point tooltips: rounding values to 5 significant figures and better formatting of dates etc.
- Addition of the `chartjs-plugin-zoom` and a "Reset Zoom" button.

6. Add `commit-msg-append` option , useful for adding e.g. `[ci skip]` to commit message, but not having it as part of the test suite key in the data JSON.
7. Removed capture of commit author/committer, since it can be obtained from the commit id/url, and just bloats the data JSON.
8. Renamed `max-items-in-chart` -> `max-data-items`, to better describe its function of truncating the saved data during a commit.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.