bencheeorg / bencheeorg/benchee
Property based benchmarking
- Dominant language
- Elixir
- Stars
- 1.5k
- Forks
- 70
- PR merge metrics
- No merged PRs in 30d
Description
This had been in my mind for a long time but I guess it's worth getting out there:
We already have inputs. Property based testing is great and support in the elixir system is growing a lot. Why not just put in a generator as an input and let it generate inputs? Statistics will become more interesting!
As generators usually generate more complex data as it goes on it might be a bit weird as the first recorded run times will be fast and the last recorded run times (or memory measurements) will be higher. We might even need a different type of generator - only time will tell.
We need to save and show the seed to ensure reproducibility.
Once we have that there are a lot of optional things we could do:
* Remember the slowest input and report it back...
* ... attempt to shrink that worst case input?
* maybe even record the statistics per input (and do each one for some time to have actual statistics)
* we probably also need to record how many iterations were run and be able to rerun them with the same number of iterations (as afterwards the generators produce new values).
* does this live in core or in a plugin? Properly cause it's an external dependency (stream data isn't core elixir yet afaik)
This is definitely a >= 1.x or 2.x feature even ;)
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.