Need a troubleshooting guide for actor developers

There is no troubleshooting documentation for actor developers.

`docs/observability.md` is good, but it answers a different question. It is **capability-oriented**: "here is the logging model, here are the metrics we emit, here is how to reach Jaeger." A developer whose actor is stuck in `STATUS_SUSPENDING` at 2am does not need a model; they need"start here, run this, and here is what the output means."

This issue asks for a **symptom-driven guide**: start from what the developer observes, route them to the right signal and tool, and say what to do with the answer.

The second deliverable matters as much as the first. Writing this guide honestly means walking every path end-to-end, and several paths dead-end today. **Those dead ends become a gap list derived from real user tasks rather than from architecture opinion**, which is a much better basis for prioritizing observability work than a wish list.

---

## Who this is for

Worth stating explicitly in the doc, because the answers differ:

| | Actor developer | Substrate operator |
|---|---|---|
| Owns | their application code and its ActorTemplate | the fleet, worker pools, nodes |
| Cares about | *"why is **my** actor misbehaving?"* | *"why is the platform degraded?"* |
| Cluster access | limited or none | full |
| Placement (pod/node) | can't act on it | primary lever |

This guide is for the first column. An operator runbook is a separate document, and conflating them is how observability docs become unusable for both.

---

## What actor developers actually have today

Grounding the guide in what exists, not what we intend to build:

| Tool / signal | Command or location | Reality check |
|---|---|---|
| Actor logs (live) | `kubectl ate logs actors -a [-f]` | Streams the **worker pod's** log, client-side filtered to the actor by `ate.dev/actor_name` + `ate.dev/actor_atespace` (`logs_actors.go:309-346`) |
| Actor logs (history) | logging backend, query on `labels."ate.dev/actor_name"` | Four documented query dimensions in `docs/observability.md` |
| Log metadata | `ate.dev/actor_{name,atespace,uid}`, `ate.dev/actor_template_{name,namespace}`, `ate.dev/container_name` | Injected by Substrate into container logs |
| Actor status | `kubectl ate get actors` | |
| Resource usage | `kubectl ate top workers` | **Workers only.** There is no `top actors` — `top.go` has exactly one subcommand |
| System metrics | `rpc.server.call.duration`, `atenet.router.route.duration`, `atelet.snapshot.size` | Three total, all platform-scoped; none carries actor identity |
| Traces | `--trace` on CLI commands → Jaeger (kind) / Cloud Trace (GKE) | Control-plane operations |
| Local UIs | `kubectl port-forward -n otel-system svc/prometheus 9090` / `svc/jaeger 16686` | kind only |
| Dashboards | `tools/setup-gcp/dashboards/` | Three, all system-facing: e2e-latency, grpc, snapshot. **None actor-facing.** GCP only |

## Proposed structure — symptom first

One section per symptom, each with the same shape: **what you'd see → which signal answers it → the command → how to read the output → what to do next.**

| Symptom | Primary signal today | Status |
|---|---|---|
| Actor won't start / stuck in a status | `get actors`, actor logs | works |
| `kubectl ate logs` says "not currently running" | — | **needs explaining, not fixing** — see below |
| Can't find logs from before a resume | backend query on `ate.dev/actor_uid` | works, but only if a backend is wired up |
| Actor is slow to first response | — | **gap**: cold-start (resume) vs. warm latency is not separable by an actor developer |
| Actor using too much memory / OOM | — | **gap** (#550) |
| Custom metrics/traces not appearing | — | **gap** (#563, #569) |
| Custom telemetry has gaps around suspend/resume | — | **known bug** (#503) |
| Actor can't reach an external service | — | **gap**: no actor-visible egress/DNS signal |
| Requests to the actor fail or hang | `atenet.router.route.duration` | operator-only, and not collected at all in kind |