agent-substrate / agent-substrate/substrate

Need an operations runbook for Substrate operators

Aperta
#581 2 commenti 0 reazioni 1 assegnatario Rivendicata da @git286 Vedi su GitHub
area/observability kind/docs prio/P0
Lingua principale
Go
Stelle
1.8k
Fork
316
Merge medio
2g 43m
PR unite (30g)
287

Descrizione

---

## Summary

There is no operational documentation for the people running Substrate.

An operator's questions are structurally different from an actor developer's. They are **fleet-level and cross-component**: is the platform healthy, which component owns this symptom, are we out of capacity, is one tenant hurting everyone, is a node degrading. Answering those means knowing which of seven-plus components to look at, and Substrate currently gives almost no help with that routing.

This issue asks for `docs/operations.md`: a symptom → component → signal → action runbook.

---

## Scope boundary

This is the operator column of the audience split; the actor-developer guide is the other. They must stay separate documents.

| | Substrate operator | Actor developer |
|---|---|---|
| Unit of concern | fleet, pools, nodes, components | one actor |
| Placement (pod/node) | **primary diagnostic lever** | not exposed — see #569 |
| Cluster access | full | limited or none |
| Typical entry point | an alert, or a tenant complaint | their own app misbehaving |

The overlap — "a tenant says their actor is slow, is it us or them?" — is the escalation boundary, and deserves its own section rather than being split across both docs.

---

## What an operator has today

| Capability | What exists | Coverage |
|---|---|---|
| Metrics | `rpc.server.call.duration` (ateapi, atelet), `atenet.router.route.duration`, `atelet.snapshot.size` | **three instruments, three components** |
| Dashboards | `ate-grpc`, `ate-e2e-latency`, `ate-snapshot` (`tools/setup-gcp/dashboards/`) | GCP only; no kind equivalent |
| Alerts | — | **none ship.** `tools/setup-gcp` has `create dashboards`; there is no `create alerts` |
| GKE scrape config | one `PodMonitoring`: `atenet-router-monitoring.yaml` | needed for the Envoy round-trip lines |
| kind scrape annotations | `prometheus.io/scrape` on ateapi, atelet, atenet-router | 3 of 7+ workloads |
| Fleet CLI | `kubectl ate get workers`, `get actors`, `get atespaces`, `top workers` | |
| Traces | `--trace`, Jaeger (kind) / Cloud Trace (GKE) | control-plane operations |
| Bounded metric labels | `ate.workerpool.name`, `ate.worker.state`, `ate.sandbox.class` (`internal/ateattr/ateattr.go:50-56`) | see the enum problem below |

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.