haskell / haskell/criterion

Add a variant of runBenchmark which also takes iteration range.

Open
#254 2 comments 0 reactions 0 assignees View on GitHub
Enhancement
Dominant language
Haskell
Stars
523
Forks
89
PR merge metrics
No merged PRs in 30d

Description

Currently there is:

```haskell
runBenchmark :: Benchmarkable -> Double -> IO (Vector Measured, Double)
```

> Run a single benchmark, and return measurements collected while executing it, along with the amount of time the measurement process took.

> Double argument is: Lower bound on how long the benchmarking process should take. In practice, this time limit may be exceeded in order to generate enough data to perform meaningful statistical analyses.

I'd like to have a variant where we can specify the optional lower and upper bounds for iteration count (run at least 10 times, no need to run over 100 times).

Lower bound would be great to ensure more statistical significance. Upper bound would allow to avoid running individual benchmark for too long.

I think that then we'd need both lower and upper bound for time as well (current is actually more like soft upper bound, but as there are no end criteria it's also a lower bound).

I think it would best to put these four options into a record, to avoid double-blindness.
In case when no bounds is specified, we could either fail or use some default time bound.

EDIT: summary:
- the end criteria would be in N dimensions: you can stop only when all lower bounds are crossed and at least one upper bound.
- no lower bound specified: zero
- no upper bounds: set time to default, other unset ones are "infinity"/unreachable.

EDIT2; `tasty-bench` does analysis, and decides end criteria on the fly. That's another option and can be used to implement the above; but i think the above is still conseptually simpler.

i offer to write a patch.

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names runBenchmark but does not identify implementation files or tests. Start by locating runBenchmark and reviewing how its current time bound is handled. Define the intended record fields and stopping criteria from the summary, then add coverage showing that iteration and time bounds are applied as specified.

Written by the indexing model from the issue text.

Assessment

Tech stack
haskell
Domain
performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.