agent-substrate / agent-substrate/substrate
[benchmarking] Capture cluster hardware density frontiers and Prometheus server telemetry in Locust runner
- Langage dominant
- Go
- Étoiles
- 1.8k
- Forks
- 316
- Merge moyen
- 2 j 43 min
- PR mergées (30 j)
- 287
Description
### Background & Problem
According to the [Actor Density Benchmark Specs](https://docs.google.com/document/d/1sv5aBXvGOQ69iaqFxdPZ-d6tNbh5AMuwKODDyYjzYQw/edit?tab=t.0), key evaluation targets are active actors per node, per vCPU, per GB RAM, and the **actor-to-pod (A/P) bin-packing ratio** (defined as **active concurrent actors / worker pod count**).
Currently:
1. `benchmarking/locust/runner.py` only records client-side request latencies and does not discover cluster hardware capacity (nodes, vCPUs, RAM allocatable).
2. During high-density benchmark runs, client requests may appear successful while server-side actors are queued or experiencing memory pressure. We need ground-truth telemetry from Prometheus (concurrent active/suspended actors, PSI pressure stalls, snapshot sizes, and ateapi throughput) captured alongside client metrics.
### Proposed Changes
1. **Cluster Hardware Discovery:** Add RBAC in `locust.yaml` allowing the runner to query `nodes` and compute cluster density frontiers (`actors/node`, `actors/vCPU`, `actors/GB RAM`, and the steady-state **A/P ratio**: `active_actors / worker_pods`) written into `stats.jsonl`.
2. **Prometheus Telemetry Harvester (`server_telemetry.py`):** Query in-cluster Prometheus at the end of benchmark trials for ground-truth actor packing, PSI kernel pressure, and snapshot performance, outputting to `server_summary.json` and `stats.jsonl`.
3. Preserve `status.json` contract (`{"locust_exit_code": 0, "stats_generated": true}`) to maintain compatibility with test harnesses and CI orchestrators.
### References
- [Agent Substrate: Actor Density Benchmark Specs](https://docs.google.com/document/d/1sv5aBXvGOQ69iaqFxdPZ-d6tNbh5AMuwKODDyYjzYQw/edit?tab=t.0)
- [Substrate Benchmark Telemetry & Metrics Enhancement](https://docs.google.com/document/d/1f9b1mTyxICeLZPojrxCujZ16Elh4IFo3_WmBeNCvk1c/edit?pli=1&tab=t.0)
cc @maxsmythe @roycaihw @aditya-shantanu
Guide de contribution
Ouvrir le guide de contribution
Évaluation
Cette issue n'a pas encore été évaluée.