agent-substrate / agent-substrate/substrate

[benchmarking] Capture cluster hardware density frontiers and Prometheus server telemetry in Locust runner

Open
#1,590 0 comments 0 reactions 0 assignees View on GitHub
area/benchmarking area/observability kind/feature
Dominant language
Go
Stars
1.8k
Forks
316
Avg merge
2d 43m
Merged PRs (30d)
287

Description

### Background & Problem
According to the [Actor Density Benchmark Specs](https://docs.google.com/document/d/1sv5aBXvGOQ69iaqFxdPZ-d6tNbh5AMuwKODDyYjzYQw/edit?tab=t.0), key evaluation targets are active actors per node, per vCPU, per GB RAM, and the **actor-to-pod (A/P) bin-packing ratio** (defined as **active concurrent actors / worker pod count**).

Currently:
1. `benchmarking/locust/runner.py` only records client-side request latencies and does not discover cluster hardware capacity (nodes, vCPUs, RAM allocatable).
2. During high-density benchmark runs, client requests may appear successful while server-side actors are queued or experiencing memory pressure. We need ground-truth telemetry from Prometheus (concurrent active/suspended actors, PSI pressure stalls, snapshot sizes, and ateapi throughput) captured alongside client metrics.

### Proposed Changes
1. **Cluster Hardware Discovery:** Add RBAC in `locust.yaml` allowing the runner to query `nodes` and compute cluster density frontiers (`actors/node`, `actors/vCPU`, `actors/GB RAM`, and the steady-state **A/P ratio**: `active_actors / worker_pods`) written into `stats.jsonl`.
2. **Prometheus Telemetry Harvester (`server_telemetry.py`):** Query in-cluster Prometheus at the end of benchmark trials for ground-truth actor packing, PSI kernel pressure, and snapshot performance, outputting to `server_summary.json` and `stats.jsonl`.
3. Preserve `status.json` contract (`{"locust_exit_code": 0, "stats_generated": true}`) to maintain compatibility with test harnesses and CI orchestrators.

### References
- [Agent Substrate: Actor Density Benchmark Specs](https://docs.google.com/document/d/1sv5aBXvGOQ69iaqFxdPZ-d6tNbh5AMuwKODDyYjzYQw/edit?tab=t.0)
- [Substrate Benchmark Telemetry & Metrics Enhancement](https://docs.google.com/document/d/1f9b1mTyxICeLZPojrxCujZ16Elh4IFo3_WmBeNCvk1c/edit?pli=1&tab=t.0)

cc @maxsmythe @roycaihw @aditya-shantanu

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.