benchmark-action / benchmark-action/github-action-benchmark

Catch2 store fails if estimated time exceeds 1m

Open
#115 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
1.3k
Forks
184
PR merge metrics
No merged PRs in 30d

Description

Using the catch2 framework to benchmark some tests I encountered the following issue:
If your total estimated time for the tests exceeds the 60s, then the display time switches to minutes. If this happens, then the store action fails.

I have adjusted the catch2 example in https://github.com/phbasler/github-action-benchmark/tree/master/examples/catch2 ( added a 0.7s sleep before the fib(10) call so that the 100 executions exceed the 60s.

In the CI workflow the "Build and run benchmarks with Catch2" part is successful, but the "Store benchmark results" sections yields the following error:

`2022-04-05T07:01:00.8358638Z ##[group]Run benchmark-action/github-action-benchmark@v1
2022-04-05T07:01:00.8358959Z with:
2022-04-05T07:01:00.8359184Z name: Catch2 Benchmark
2022-04-05T07:01:00.8359428Z tool: catch2
2022-04-05T07:01:00.8359724Z output-file-path: examples/catch2/benchmark_result.txt
2022-04-05T07:01:00.8360710Z github-token: ***
2022-04-05T07:01:00.8360988Z auto-push: true
2022-04-05T07:01:00.8361230Z alert-threshold: 200%
2022-04-05T07:01:00.8361470Z comment-on-alert: true
2022-04-05T07:01:00.8361716Z fail-on-alert: true
2022-04-05T07:01:00.8401114Z alert-comment-cc-users: @bernedom,@ktrz
2022-04-05T07:01:00.8401461Z gh-pages-branch: gh-pages
2022-04-05T07:01:00.8401784Z benchmark-data-dir-path: dev/bench
2022-04-05T07:01:00.8402088Z skip-fetch-gh-pages: false
2022-04-05T07:01:00.8402344Z comment-always: false
2022-04-05T07:01:00.8402594Z save-data-file: true
2022-04-05T07:01:00.8402834Z ##[endgroup]
2022-04-05T07:01:01.1730005Z ##[error]No benchmark found for bench suite. Possibly mangled output from Catch2:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Catch2_bench is a Catch v2.11.0 host application.
Run with -? for options

-------------------------------------------------------------------------------
Fibonacci
-------------------------------------------------------------------------------
/home/runner/work/github-action-benchmark/github-action-benchmark/examples/catch2/catch2_bench.cpp:8
...............................................................................

benchmark name samples iterations estimated
mean low mean high mean
std dev low std dev high std dev
-------------------------------------------------------------------------------
Fibonacci 10 100 1 1.16689 m
700.121 ms 700.118 ms 700.124 ms
16.061 us 14.141 us 18.543 us

Fibonacci 20 100 2 6.051 ms
30.137 us 29.714 us 32.152 us
4.019 us 80 ns 9.574 us

===============================================================================
test cases: 1 | 1 passed
assertions: - none -`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.