bytecodealliance / bytecodealliance/sightglass

Detect and warn if the samples are not normally distributed

Open
#142 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
WebAssembly
Stars
88
Forks
37
Avg merge
5h 5m
Merged PRs (30d)
1

Description

From #138:

> Ah, and one more thought: have we considered any statistical analysis that would look for multi-modal distributions (and warn, at least)? If we see that e.g. half of all runs of a benchmark run in 0.3s and half in 0.5s, and the distribution looks like the sum of two Gaussians, it may be better to warn the user "please check settings X, Y, Z; you seem to be alternating between two different configurations randomly" than to just present a mean of 0.4s with some wide variance, while the latter makes more sense if we just have a single Gaussian with truly random noise.

We can use the Shapiro-Wilk test to determine normality: https://en.wikipedia.org/wiki/Shapiro%E2%80%93Wilk_test

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.