awslabs / awslabs/llmeter

Add a built-in profiling callback for per-invocation timing analysis

Open
#90 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
41
Forks
5
Avg merge
2m
Merged PRs (30d)
1

Description

## Problem

When investigating where time is spent during an LLMeter run, users currently need to write custom ad-hoc callbacks (as seen in dev notebooks) to capture per-request phase timings. There is no built-in way to:

1. **Break down request time into server-side phases** — specifically separating prefill (TTFT) from generation (decode), and distinguishing server time from client-side runner overhead.

2. **Capture per-invocation metadata** alongside timing — cache hit status, reasoning token counts, retry counts, and throughput metrics — in a structured format suitable for analysis.

3. **Persist profiling data** to disk for post-hoc analysis without needing the live callback or endpoint.

4. **Visualize profiling results** with ready-made plots (phase breakdown, request timeline/concurrency, throughput over time, TPOT distribution, etc.).

## Expected Behavior

A built-in `ProfileCallback` that:
- Hooks into the existing callback lifecycle (`before_invoke` / `after_invoke` / `after_run`)
- Captures detailed per-invocation profiles (timing phases, token counts, cache info, retries)
- Produces an aggregated report with time accounting and percentile statistics
- Saves structured data (JSON + JSONL) alongside run results
- Contributes summary stats to `result.stats` for programmatic access
- Pairs with plotting utilities for common profiling visualizations

## Use Case

Understanding where wall-clock time goes in a benchmark run — especially distinguishing server-side prefill and decode from client-side overhead — and identifying patterns like cache hit impact on TTFT, throughput degradation over time, or concurrency effects.

Contributor guide

Open the contributing guide

Research direction

Start with the existing callback lifecycle entry points: before_invoke, after_invoke, and after_run. Define how per-invocation timings, token counts, cache information, retries, and throughput feed the aggregated report, result.stats, and JSON/JSONL outputs. Done includes the requested plotting utilities and persisted profiling data for phase, timeline, throughput, and TPOT analysis.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.