agent-substrate / agent-substrate/substrate

Need an operations runbook for Substrate operators

Đang mở
#581 2 bình luận 0 reaction 1 người được giao Được @git286 nhận Xem trên GitHub
area/observability kind/docs prio/P0
Ngôn ngữ chính
Go
Star
1.8k
Fork
316
Merge trung bình
2 ngày 43 phút
Pull request đã merge (30 ngày)
287

Mô tả

---

## Summary

There is no operational documentation for the people running Substrate.

An operator's questions are structurally different from an actor developer's. They are **fleet-level and cross-component**: is the platform healthy, which component owns this symptom, are we out of capacity, is one tenant hurting everyone, is a node degrading. Answering those means knowing which of seven-plus components to look at, and Substrate currently gives almost no help with that routing.

This issue asks for `docs/operations.md`: a symptom → component → signal → action runbook.

---

## Scope boundary

This is the operator column of the audience split; the actor-developer guide is the other. They must stay separate documents.

| | Substrate operator | Actor developer |
|---|---|---|
| Unit of concern | fleet, pools, nodes, components | one actor |
| Placement (pod/node) | **primary diagnostic lever** | not exposed — see #569 |
| Cluster access | full | limited or none |
| Typical entry point | an alert, or a tenant complaint | their own app misbehaving |

The overlap — "a tenant says their actor is slow, is it us or them?" — is the escalation boundary, and deserves its own section rather than being split across both docs.

---

## What an operator has today

| Capability | What exists | Coverage |
|---|---|---|
| Metrics | `rpc.server.call.duration` (ateapi, atelet), `atenet.router.route.duration`, `atelet.snapshot.size` | **three instruments, three components** |
| Dashboards | `ate-grpc`, `ate-e2e-latency`, `ate-snapshot` (`tools/setup-gcp/dashboards/`) | GCP only; no kind equivalent |
| Alerts | — | **none ship.** `tools/setup-gcp` has `create dashboards`; there is no `create alerts` |
| GKE scrape config | one `PodMonitoring`: `atenet-router-monitoring.yaml` | needed for the Envoy round-trip lines |
| kind scrape annotations | `prometheus.io/scrape` on ateapi, atelet, atenet-router | 3 of 7+ workloads |
| Fleet CLI | `kubectl ate get workers`, `get actors`, `get atespaces`, `top workers` | |
| Traces | `--trace`, Jaeger (kind) / Cloud Trace (GKE) | control-plane operations |
| Bounded metric labels | `ate.workerpool.name`, `ate.worker.state`, `ate.sandbox.class` (`internal/ateattr/ateattr.go:50-56`) | see the enum problem below |

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.