bencheeorg / bencheeorg/benchee

Property based benchmarking

Open
#249 6 comments 2 reactions 0 assignees View on GitHub
enhancement
Dominant language
Elixir
Stars
1.5k
Forks
70
PR merge metrics
No merged PRs in 30d

Description

This had been in my mind for a long time but I guess it's worth getting out there:

We already have inputs. Property based testing is great and support in the elixir system is growing a lot. Why not just put in a generator as an input and let it generate inputs? Statistics will become more interesting!

As generators usually generate more complex data as it goes on it might be a bit weird as the first recorded run times will be fast and the last recorded run times (or memory measurements) will be higher. We might even need a different type of generator - only time will tell.

We need to save and show the seed to ensure reproducibility.

Once we have that there are a lot of optional things we could do:

* Remember the slowest input and report it back...
* ... attempt to shrink that worst case input?
* maybe even record the statistics per input (and do each one for some time to have actual statistics)
* we probably also need to record how many iterations were run and be able to rerun them with the same number of iterations (as afterwards the generators produce new values).
* does this live in core or in a plugin? Properly cause it's an external dependency (stream data isn't core elixir yet afaik)

This is definitely a >= 1.x or 2.x feature even ;)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.