Add a variant of runBenchmark which also takes iteration range.
- Dominant language
- Haskell
- Stars
- 523
- Forks
- 89
- PR merge metrics
- No merged PRs in 30d
Description
Currently there is:
```haskell
runBenchmark :: Benchmarkable -> Double -> IO (Vector Measured, Double)
```
> Run a single benchmark, and return measurements collected while executing it, along with the amount of time the measurement process took.
> Double argument is: Lower bound on how long the benchmarking process should take. In practice, this time limit may be exceeded in order to generate enough data to perform meaningful statistical analyses.
I'd like to have a variant where we can specify the optional lower and upper bounds for iteration count (run at least 10 times, no need to run over 100 times).
Lower bound would be great to ensure more statistical significance. Upper bound would allow to avoid running individual benchmark for too long.
I think that then we'd need both lower and upper bound for time as well (current is actually more like soft upper bound, but as there are no end criteria it's also a lower bound).
I think it would best to put these four options into a record, to avoid double-blindness.
In case when no bounds is specified, we could either fail or use some default time bound.
EDIT: summary:
- the end criteria would be in N dimensions: you can stop only when all lower bounds are crossed and at least one upper bound.
- no lower bound specified: zero
- no upper bounds: set time to default, other unset ones are "infinity"/unreachable.
EDIT2; `tasty-bench` does analysis, and decides end criteria on the fly. That's another option and can be used to implement the above; but i think the above is still conseptually simpler.
i offer to write a patch.
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue names runBenchmark but does not identify implementation files or tests. Start by locating runBenchmark and reviewing how its current time bound is handled. Define the intended record fields and stopping criteria from the summary, then add coverage showing that iteration and time bounds are applied as specified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- haskell
- Domain
- performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100