agent-substrate / agent-substrate/substrate
Need an operations runbook for Substrate operators
- 主要語言
- Go
- 星號
- 1.8k
- 分支
- 316
- 平均合併
- 2 天 43 分鐘
- 30 天內合併 PR
- 287
描述
---
## Summary
There is no operational documentation for the people running Substrate.
An operator's questions are structurally different from an actor developer's. They are **fleet-level and cross-component**: is the platform healthy, which component owns this symptom, are we out of capacity, is one tenant hurting everyone, is a node degrading. Answering those means knowing which of seven-plus components to look at, and Substrate currently gives almost no help with that routing.
This issue asks for `docs/operations.md`: a symptom → component → signal → action runbook.
---
## Scope boundary
This is the operator column of the audience split; the actor-developer guide is the other. They must stay separate documents.
| | Substrate operator | Actor developer |
|---|---|---|
| Unit of concern | fleet, pools, nodes, components | one actor |
| Placement (pod/node) | **primary diagnostic lever** | not exposed — see #569 |
| Cluster access | full | limited or none |
| Typical entry point | an alert, or a tenant complaint | their own app misbehaving |
The overlap — "a tenant says their actor is slow, is it us or them?" — is the escalation boundary, and deserves its own section rather than being split across both docs.
---
## What an operator has today
| Capability | What exists | Coverage |
|---|---|---|
| Metrics | `rpc.server.call.duration` (ateapi, atelet), `atenet.router.route.duration`, `atelet.snapshot.size` | **three instruments, three components** |
| Dashboards | `ate-grpc`, `ate-e2e-latency`, `ate-snapshot` (`tools/setup-gcp/dashboards/`) | GCP only; no kind equivalent |
| Alerts | — | **none ship.** `tools/setup-gcp` has `create dashboards`; there is no `create alerts` |
| GKE scrape config | one `PodMonitoring`: `atenet-router-monitoring.yaml` | needed for the Envoy round-trip lines |
| kind scrape annotations | `prometheus.io/scrape` on ateapi, atelet, atenet-router | 3 of 7+ workloads |
| Fleet CLI | `kubectl ate get workers`, `get actors`, `get atespaces`, `top workers` | |
| Traces | `--trace`, Jaeger (kind) / Cloud Trace (GKE) | control-plane operations |
| Bounded metric labels | `ate.workerpool.name`, `ate.worker.state`, `ate.sandbox.class` (`internal/ateattr/ateattr.go:50-56`) | see the enum problem below |
貢獻指南
評估
這個 Issue 還沒有評估資料。