agent-substrate / agent-substrate/substrate

[benchmarking] Capture cluster hardware density frontiers and Prometheus server telemetry in Locust runner

未關閉
#1,590 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
area/benchmarking area/observability kind/feature
主要語言
Go
星號
1.8k
分支
316
平均合併
2 天 43 分鐘
30 天內合併 PR
287

描述

### Background & Problem
According to the [Actor Density Benchmark Specs](https://docs.google.com/document/d/1sv5aBXvGOQ69iaqFxdPZ-d6tNbh5AMuwKODDyYjzYQw/edit?tab=t.0), key evaluation targets are active actors per node, per vCPU, per GB RAM, and the **actor-to-pod (A/P) bin-packing ratio** (defined as **active concurrent actors / worker pod count**).

Currently:
1. `benchmarking/locust/runner.py` only records client-side request latencies and does not discover cluster hardware capacity (nodes, vCPUs, RAM allocatable).
2. During high-density benchmark runs, client requests may appear successful while server-side actors are queued or experiencing memory pressure. We need ground-truth telemetry from Prometheus (concurrent active/suspended actors, PSI pressure stalls, snapshot sizes, and ateapi throughput) captured alongside client metrics.

### Proposed Changes
1. **Cluster Hardware Discovery:** Add RBAC in `locust.yaml` allowing the runner to query `nodes` and compute cluster density frontiers (`actors/node`, `actors/vCPU`, `actors/GB RAM`, and the steady-state **A/P ratio**: `active_actors / worker_pods`) written into `stats.jsonl`.
2. **Prometheus Telemetry Harvester (`server_telemetry.py`):** Query in-cluster Prometheus at the end of benchmark trials for ground-truth actor packing, PSI kernel pressure, and snapshot performance, outputting to `server_summary.json` and `stats.jsonl`.
3. Preserve `status.json` contract (`{"locust_exit_code": 0, "stats_generated": true}`) to maintain compatibility with test harnesses and CI orchestrators.

### References
- [Agent Substrate: Actor Density Benchmark Specs](https://docs.google.com/document/d/1sv5aBXvGOQ69iaqFxdPZ-d6tNbh5AMuwKODDyYjzYQw/edit?tab=t.0)
- [Substrate Benchmark Telemetry & Metrics Enhancement](https://docs.google.com/document/d/1f9b1mTyxICeLZPojrxCujZ16Elh4IFo3_WmBeNCvk1c/edit?pli=1&tab=t.0)

cc @maxsmythe @roycaihw @aditya-shantanu

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。