Statistical soundness of removing outliers from a heavy tailed distribution?

Latency values (such as those from performance time measurements) often follow a heavy-tailed distribution -- especially when there are spontaneous delays involved, e.g. when garbage collection enters the picture.

It is a common mistake to assume a normal distribution for latency/performance measurements. Sometimes this normality assumption holds, but this is not a safe assumption without testing it first.

The central limit theorem helps, of course, but severely fat-tailed distributions only converge to a normal distribution very, very slowly. This means that even when looking at a sum of measurements (which is the case when computing the mean, of course) normality should not be blindly assumed.

Under a heavy-tailed distribution, the sample mean (and even the high centiles) tend to underestimate the corresponding true values for the population. Imagine my surprise when I run benchmarking of a computation that has clearly heavy-tailed running times, and BenchmarkDotNet reports the sample mean as the primary measure of interest. To make matters worse, it reports outliers have been discarded! The very values that are supposed to at least help a tiny bit in reducing the underestimation -- removed!

When dealing with heavy-tailed distributions, the majority of samples don't tell us a whole lot about the true underlying distribution. The strongest signal can be found in the "outliers" -- these should not be ignored; if anything, they're what I would focus on! I can't in good faith use whatever values BenchmarkDotNet keep when it reports having ignored the strongest signal.

In my case, BenchmarkDotNet was friendly enough to report what range the removed outliers were in, so I could report that value to my co-worker.

However, if BenchmarkDotNet does not verify normality before computing means and standard deviations (beyond just guessing an *n* that "should" suffice) it seems misleading to emphasise these values; especially considering the position of BenchmarkDotNet as a black box tool (that in all other ways is VERY GOOD in that regard.)

**My suggestion in this issue is emphasising some other measure in the cases where normality cannot be confirmed.**

A simple and more robust measure is the maximum value, but I'm interested in what other alternatives there are to consider.