bytecodealliance / bytecodealliance/sightglass
Detect and warn if the samples are not normally distributed
- Dominant language
- WebAssembly
- Stars
- 88
- Forks
- 37
- Avg merge
- 5h 5m
- Merged PRs (30d)
- 1
Description
From #138:
> Ah, and one more thought: have we considered any statistical analysis that would look for multi-modal distributions (and warn, at least)? If we see that e.g. half of all runs of a benchmark run in 0.3s and half in 0.5s, and the distribution looks like the sum of two Gaussians, it may be better to warn the user "please check settings X, Y, Z; you seem to be alternating between two different configurations randomly" than to just present a mean of 0.4s with some wide variance, while the latter makes more sense if we just have a single Gaussian with truly random noise.
We can use the Shapiro-Wilk test to determine normality: https://en.wikipedia.org/wiki/Shapiro%E2%80%93Wilk_test
Contributor guide
Assessment
This issue has not been assessed yet.