awslabs / awslabs/llmeter

Feature: System metrics monitoring (CPU, memory, network I/O) during benchmark runs

Open
#83 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
41
Forks
5
Avg merge
2m
Merged PRs (30d)
1

Description

## Motivation

When running latency benchmarks — especially load tests with high concurrency — it's important to know whether the client machine itself is a bottleneck. High CPU usage, memory pressure, or network saturation on the client side can skew latency measurements without any visible errors.

Currently there's no built-in way to observe client-side resource usage during a run.

## Proposed feature

Add an optional `SystemMetricsMonitor` callback that tracks CPU, memory, and network I/O while a benchmark runs. It should:

- Collect periodic samples of system resource usage during the run
- Contribute aggregated statistics (avg, p50, p90, p99, max) to `result.stats`
- Show live metrics in the progress display during the run
- Persist metrics as part of `stats.json` so they survive save/load round-trips
- Be optional (gated behind an extra dependency)
- Support both per-process and system-wide monitoring modes

## Use cases

- Detecting whether the client machine is saturated during high-concurrency load tests
- Correlating network throughput with latency degradation
- Validating that benchmarks ran on a machine with sufficient headroom
- Comparing resource usage across different concurrency levels

Contributor guide

Open the contributing guide

Research direction

Start by tracing how benchmark runs produce result.stats, update the progress display, and save or load stats.json. Define how the optional SystemMetricsMonitor callback would collect per-process and system-wide samples, aggregate avg, p50, p90, p99, and max values, and expose them consistently. Done means metrics are visible during runs, persist through save/load round-trips, and remain optional behind an extra dependency.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.