Estimate uncertainty of quantile/stats
- Dominant language
- Python
- Stars
- 41
- Forks
- 5
- Avg merge
- 2m
- Merged PRs (30d)
- 1
Description
Currently, LLMeter provides point estimates of quantile stats (p90, p99, etc) and general warnings in the docs about the unreliability of extreme quantiles for small sample sizes.
Maybe we could consider providing more robust estimates of this uncertainty, to help guide when the quantiles aren't appropriate to use?
For example there's a proposed [implementation here](https://skorski.info/2024/05/29/correct-confidence-intervals-for-quantiles/) of confidence interval estimation for a quantile, and Claude suggested a similar approach as below:
```python
from scipy.stats import binom
def quantile_ci(data, p, confidence=0.95):
sorted_data = sorted(data)
n = len(sorted_data)
alpha = 1 - confidence
# Find lower and upper order statistic indices
lower = binom.ppf(alpha / 2, n, p) # e.g. index of CI lower bound
upper = binom.ppf(1 - alpha / 2, n, p)
l, u = int(lower), int(min(upper, n - 1))
if l == u:
return None # Can't form a CI — sample too small for this quantile
return (sorted_data[l], sorted_data[u])
```
I'd need to spend longer staring at the stats to fully validate, but the general gist seems to be that there should be possibilities - with the main dependency being some kind of representation of the PMF/CDF of Binomial distribution... Which might be possible without taking on the heavy scipy library?
If we don't like the complexity of actually exposing the confidence intervals to users, then maybe it could just form the basis of an internal validity check that omits quantile stats we can't reliably estimate given the collected sample set?
Contributor guide
Research direction
Start by reviewing how LLMeter currently computes and exposes p90/p99 and the documentation warnings about small sample sizes. Validate the proposed order-statistic and binomial confidence-interval approach against the linked article, and assess whether a lightweight PMF/CDF implementation is preferable to scipy. Done means the project has a decided, validated approach for handling unreliable quantiles.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100