Add a built-in profiling callback for per-invocation timing analysis
- Dominant language
- Python
- Stars
- 41
- Forks
- 5
- Avg merge
- 2m
- Merged PRs (30d)
- 1
Description
## Problem
When investigating where time is spent during an LLMeter run, users currently need to write custom ad-hoc callbacks (as seen in dev notebooks) to capture per-request phase timings. There is no built-in way to:
1. **Break down request time into server-side phases** — specifically separating prefill (TTFT) from generation (decode), and distinguishing server time from client-side runner overhead.
2. **Capture per-invocation metadata** alongside timing — cache hit status, reasoning token counts, retry counts, and throughput metrics — in a structured format suitable for analysis.
3. **Persist profiling data** to disk for post-hoc analysis without needing the live callback or endpoint.
4. **Visualize profiling results** with ready-made plots (phase breakdown, request timeline/concurrency, throughput over time, TPOT distribution, etc.).
## Expected Behavior
A built-in `ProfileCallback` that:
- Hooks into the existing callback lifecycle (`before_invoke` / `after_invoke` / `after_run`)
- Captures detailed per-invocation profiles (timing phases, token counts, cache info, retries)
- Produces an aggregated report with time accounting and percentile statistics
- Saves structured data (JSON + JSONL) alongside run results
- Contributes summary stats to `result.stats` for programmatic access
- Pairs with plotting utilities for common profiling visualizations
## Use Case
Understanding where wall-clock time goes in a benchmark run — especially distinguishing server-side prefill and decode from client-side overhead — and identifying patterns like cache hit impact on TTFT, throughput degradation over time, or concurrency effects.
Contributor guide
Research direction
Start with the existing callback lifecycle entry points: before_invoke, after_invoke, and after_run. Define how per-invocation timings, token counts, cache information, retries, and throughput feed the aggregated report, result.stats, and JSON/JSONL outputs. Done includes the requested plotting utilities and persisted profiling data for phase, timeline, throughput, and TPOT analysis.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100