awslabs / awslabs/llmeter

Estimate uncertainty of quantile/stats

Open
#59 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
41
Forks
5
Avg merge
2m
Merged PRs (30d)
1

Description

Currently, LLMeter provides point estimates of quantile stats (p90, p99, etc) and general warnings in the docs about the unreliability of extreme quantiles for small sample sizes.

Maybe we could consider providing more robust estimates of this uncertainty, to help guide when the quantiles aren't appropriate to use?

For example there's a proposed [implementation here](https://skorski.info/2024/05/29/correct-confidence-intervals-for-quantiles/) of confidence interval estimation for a quantile, and Claude suggested a similar approach as below:

```python
from scipy.stats import binom

def quantile_ci(data, p, confidence=0.95):
sorted_data = sorted(data)
n = len(sorted_data)
alpha = 1 - confidence
# Find lower and upper order statistic indices
lower = binom.ppf(alpha / 2, n, p) # e.g. index of CI lower bound
upper = binom.ppf(1 - alpha / 2, n, p)
l, u = int(lower), int(min(upper, n - 1))
if l == u:
return None # Can't form a CI — sample too small for this quantile
return (sorted_data[l], sorted_data[u])
```

I'd need to spend longer staring at the stats to fully validate, but the general gist seems to be that there should be possibilities - with the main dependency being some kind of representation of the PMF/CDF of Binomial distribution... Which might be possible without taking on the heavy scipy library?

If we don't like the complexity of actually exposing the confidence intervals to users, then maybe it could just form the basis of an internal validity check that omits quantile stats we can't reliably estimate given the collected sample set?

Contributor guide

Open the contributing guide

Research direction

Start by reviewing how LLMeter currently computes and exposes p90/p99 and the documentation warnings about small sample sizes. Validate the proposed order-statistic and binomial confidence-interval approach against the linked article, and assess whether a lightweight PMF/CDF implementation is preferable to scipy. Done means the project has a decided, validated approach for handling unreliable quantiles.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.