Do more per VM invocation
- Dominant language
- Java
- Stars
- 818
- Forks
- 112
- Avg merge
- 12m
- Merged PRs (30d)
- 5
Description
```
Suppose I'm testing 4 x 3 different parameter values against 5 different
benchmarks (different time- methods in the same class) on 2 vms.
Currently, to get one measurement each, we'll run 4x3x5x2=120 vm
invocations. I think 10 would be enough -- vms times benchmarks, and let
each run handle all 4 x 3 parameter combinations for that (vm,benchmark)
pair.
The problem with the way it is today is that hotspot can optimize away
whole swaths of implementation code that doesn't happen to get exercised by
the *one* scenario we run it with. By warming up all 12 of these benchmark
instances, it should have to compile to something more closely resembling
real life (maybe). And with luck, we can avoid the expense of repeating
the warmup period 12 times over.
After warming up all the different scenarios and then starting to do trials
of one of them, I'm not sure if we need to worry about hotspot deciding to
*re*compile based on the new favorite scenario. If that happens, maybe it
makes sense for us to round-robin through the scenarios as we go......
we'll see.
I'm also not sure how concerned we need to be that the order the scenarios
are timed in can unduly affect the results. It could be that for each
"redundant" measurement we take, we vary up the order (e.g. we rotate it?)
in order to wash that out. Or maybe there's no problem with this; I dunno.
```
Original issue reported on code.google.com by `kevinb@google.com` on 22 Jan 2010 at 10:53
Contributor guide
Assessment
This issue has not been assessed yet.