agent-substrate / agent-substrate/substrate

Need an operations runbook for Substrate operators

Abierto
#581 2 comentarios 0 reacciones 1 asignado Reclamado por @git286 Ver en GitHub
area/observability kind/docs prio/P0
Lenguaje dominante
Go
Estrellas
1.8k
Forks
316
Merge medio
2 d 43 min
PR fusionados (30 d)
287

Descripción

---

## Summary

There is no operational documentation for the people running Substrate.

An operator's questions are structurally different from an actor developer's. They are **fleet-level and cross-component**: is the platform healthy, which component owns this symptom, are we out of capacity, is one tenant hurting everyone, is a node degrading. Answering those means knowing which of seven-plus components to look at, and Substrate currently gives almost no help with that routing.

This issue asks for `docs/operations.md`: a symptom → component → signal → action runbook.

---

## Scope boundary

This is the operator column of the audience split; the actor-developer guide is the other. They must stay separate documents.

| | Substrate operator | Actor developer |
|---|---|---|
| Unit of concern | fleet, pools, nodes, components | one actor |
| Placement (pod/node) | **primary diagnostic lever** | not exposed — see #569 |
| Cluster access | full | limited or none |
| Typical entry point | an alert, or a tenant complaint | their own app misbehaving |

The overlap — "a tenant says their actor is slow, is it us or them?" — is the escalation boundary, and deserves its own section rather than being split across both docs.

---

## What an operator has today

| Capability | What exists | Coverage |
|---|---|---|
| Metrics | `rpc.server.call.duration` (ateapi, atelet), `atenet.router.route.duration`, `atelet.snapshot.size` | **three instruments, three components** |
| Dashboards | `ate-grpc`, `ate-e2e-latency`, `ate-snapshot` (`tools/setup-gcp/dashboards/`) | GCP only; no kind equivalent |
| Alerts | — | **none ship.** `tools/setup-gcp` has `create dashboards`; there is no `create alerts` |
| GKE scrape config | one `PodMonitoring`: `atenet-router-monitoring.yaml` | needed for the Envoy round-trip lines |
| kind scrape annotations | `prometheus.io/scrape` on ateapi, atelet, atenet-router | 3 of 7+ workloads |
| Fleet CLI | `kubectl ate get workers`, `get actors`, `get atespaces`, `top workers` | |
| Traces | `--trace`, Jaeger (kind) / Cloud Trace (GKE) | control-plane operations |
| Bounded metric labels | `ate.workerpool.name`, `ate.worker.state`, `ate.sandbox.class` (`internal/ateattr/ateattr.go:50-56`) | see the enum problem below |

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.