agent-substrate / agent-substrate/substrate

[benchmarking] Capture cluster hardware density frontiers and Prometheus server telemetry in Locust runner

未关闭
#1,590 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
area/benchmarking area/observability kind/feature
主要语言
Go
星标
1.8k
派生
316
平均合并
2 天 43 分钟
30 天内合并 PR
287

描述

### Background & Problem
According to the [Actor Density Benchmark Specs](https://docs.google.com/document/d/1sv5aBXvGOQ69iaqFxdPZ-d6tNbh5AMuwKODDyYjzYQw/edit?tab=t.0), key evaluation targets are active actors per node, per vCPU, per GB RAM, and the **actor-to-pod (A/P) bin-packing ratio** (defined as **active concurrent actors / worker pod count**).

Currently:
1. `benchmarking/locust/runner.py` only records client-side request latencies and does not discover cluster hardware capacity (nodes, vCPUs, RAM allocatable).
2. During high-density benchmark runs, client requests may appear successful while server-side actors are queued or experiencing memory pressure. We need ground-truth telemetry from Prometheus (concurrent active/suspended actors, PSI pressure stalls, snapshot sizes, and ateapi throughput) captured alongside client metrics.

### Proposed Changes
1. **Cluster Hardware Discovery:** Add RBAC in `locust.yaml` allowing the runner to query `nodes` and compute cluster density frontiers (`actors/node`, `actors/vCPU`, `actors/GB RAM`, and the steady-state **A/P ratio**: `active_actors / worker_pods`) written into `stats.jsonl`.
2. **Prometheus Telemetry Harvester (`server_telemetry.py`):** Query in-cluster Prometheus at the end of benchmark trials for ground-truth actor packing, PSI kernel pressure, and snapshot performance, outputting to `server_summary.json` and `stats.jsonl`.
3. Preserve `status.json` contract (`{"locust_exit_code": 0, "stats_generated": true}`) to maintain compatibility with test harnesses and CI orchestrators.

### References
- [Agent Substrate: Actor Density Benchmark Specs](https://docs.google.com/document/d/1sv5aBXvGOQ69iaqFxdPZ-d6tNbh5AMuwKODDyYjzYQw/edit?tab=t.0)
- [Substrate Benchmark Telemetry & Metrics Enhancement](https://docs.google.com/document/d/1f9b1mTyxICeLZPojrxCujZ16Elh4IFo3_WmBeNCvk1c/edit?pli=1&tab=t.0)

cc @maxsmythe @roycaihw @aditya-shantanu

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。