agent-substrate / agent-substrate/substrate

[benchmarking] Capture cluster hardware density frontiers and Prometheus server telemetry in Locust runner

Đang mở
#1,590 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
area/benchmarking area/observability kind/feature
Ngôn ngữ chính
Go
Star
1.8k
Fork
316
Merge trung bình
2 ngày 43 phút
Pull request đã merge (30 ngày)
287

Mô tả

### Background & Problem
According to the [Actor Density Benchmark Specs](https://docs.google.com/document/d/1sv5aBXvGOQ69iaqFxdPZ-d6tNbh5AMuwKODDyYjzYQw/edit?tab=t.0), key evaluation targets are active actors per node, per vCPU, per GB RAM, and the **actor-to-pod (A/P) bin-packing ratio** (defined as **active concurrent actors / worker pod count**).

Currently:
1. `benchmarking/locust/runner.py` only records client-side request latencies and does not discover cluster hardware capacity (nodes, vCPUs, RAM allocatable).
2. During high-density benchmark runs, client requests may appear successful while server-side actors are queued or experiencing memory pressure. We need ground-truth telemetry from Prometheus (concurrent active/suspended actors, PSI pressure stalls, snapshot sizes, and ateapi throughput) captured alongside client metrics.

### Proposed Changes
1. **Cluster Hardware Discovery:** Add RBAC in `locust.yaml` allowing the runner to query `nodes` and compute cluster density frontiers (`actors/node`, `actors/vCPU`, `actors/GB RAM`, and the steady-state **A/P ratio**: `active_actors / worker_pods`) written into `stats.jsonl`.
2. **Prometheus Telemetry Harvester (`server_telemetry.py`):** Query in-cluster Prometheus at the end of benchmark trials for ground-truth actor packing, PSI kernel pressure, and snapshot performance, outputting to `server_summary.json` and `stats.jsonl`.
3. Preserve `status.json` contract (`{"locust_exit_code": 0, "stats_generated": true}`) to maintain compatibility with test harnesses and CI orchestrators.

### References
- [Agent Substrate: Actor Density Benchmark Specs](https://docs.google.com/document/d/1sv5aBXvGOQ69iaqFxdPZ-d6tNbh5AMuwKODDyYjzYQw/edit?tab=t.0)
- [Substrate Benchmark Telemetry & Metrics Enhancement](https://docs.google.com/document/d/1f9b1mTyxICeLZPojrxCujZ16Elh4IFo3_WmBeNCvk1c/edit?pli=1&tab=t.0)

cc @maxsmythe @roycaihw @aditya-shantanu

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.