awslabs / awslabs/llmeter

Plots/stats/analyses for scaling dynamics

Open
#33 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
41
Forks
5
Avg merge
2m
Merged PRs (30d)
1

Description

AFAIK the plots and stats in LLMeter today all treat each Run as a bucket of IID data points to calculate statistics over - ignoring trends/dynamics during the run.

We're interested for LLMeter to provide more tools for analyzing auto-scaling responses over time to step increases in demand. For example:
1. Were there significant trends/shifts in TTFT or TTLT/TPOT or error rate over time during the Run, or was it homogeneous?
2. If so, how long did latency/error rate take to "stabilize", and could we characterize stats for that steady-state excluding the scale-up period?

Open to suggestions on good/practical ways to approach this, but it seems like a gap in the current tooling.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing LLMeter’s existing plots and statistics, focusing on how each Run is currently treated as IID data points. Define practical analyses for trends in TTFT, TTLT/TPOT, and error rate during step increases in demand. Done means reporting whether metrics stabilize, estimating stabilization time, and calculating steady-state statistics excluding scale-up.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-visualization, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.