Plots/stats/analyses for scaling dynamics
- Dominant language
- Python
- Stars
- 41
- Forks
- 5
- Avg merge
- 2m
- Merged PRs (30d)
- 1
Description
AFAIK the plots and stats in LLMeter today all treat each Run as a bucket of IID data points to calculate statistics over - ignoring trends/dynamics during the run.
We're interested for LLMeter to provide more tools for analyzing auto-scaling responses over time to step increases in demand. For example:
1. Were there significant trends/shifts in TTFT or TTLT/TPOT or error rate over time during the Run, or was it homogeneous?
2. If so, how long did latency/error rate take to "stabilize", and could we characterize stats for that steady-state excluding the scale-up period?
Open to suggestions on good/practical ways to approach this, but it seems like a gap in the current tooling.
Contributor guide
Research direction
Start by reviewing LLMeter’s existing plots and statistics, focusing on how each Run is currently treated as IID data points. Define practical analyses for trends in TTFT, TTLT/TPOT, and error rate during step increases in demand. Done means reporting whether metrics stabilize, estimating stabilization time, and calculating steady-state statistics excluding scale-up.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-visualization, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100