bencheeorg / bencheeorg/benchee

Suggestion: Option to kill long-running scenarios early

Open
#337 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
Elixir
Stars
1.5k
Forks
70
PR merge metrics
No merged PRs in 30d

Description

The case where we have a couple of different implementations and we want to test them on small, medium, large and super-large inputs is rather common.
Often, we refer to some kind of 'simple' structure as baseline, and want to benchmark a more clever 'complex' structure.

As a simple example, consider benchmarking element-access of a list using `Enum.fetch(list, index)` to that of `:array.get(array, index)`.

For small inputs, the simple structure might run fast enough (and because of its simplicity, it might actually be _faster_ than the complex structure). However, for larger inputs it will slow down dramatically. While this is nice to show in a graph, at some point we reach a problem:

- If lucky, benchmarking the larger inputs takes much longer than the 'estimated time'
- If unlucky, benchmarking the larger inputs takes so much time and/or memory that the computer freezes, or the elixir process is killed.

The only way to resolve this issue right now is to:
1. Run the benchmark, keeping a close eye.
2. If it seems to take rather long, interrupt it manually and remove the input case where the benchmarks started hanging
3. Repeat from (1) until the benchmark actually completes.

An additional drawback is that we have now removed the large inputs for all implementations. This means that if we have multiple 'clever' implementations, we can no longer see how they compare between them for large inputs unless we create another separate benchmark where the simple implementation is missing.

---

Proposal: Add an option to stop a benchmark scenario if its time to complete is dis-proportionally long w.r.t. the estimated running time. For instance, if `time: 5s`, we might stop any benchmark scenario where a single run already takes longer than `5s` to complete.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.