Add validator observability and incident response

@lucacillario is already working on this.

Since Sep 17, 2026.

### Goal

Give the team early, actionable warning when validator health degrades and enough context to diagnose and resolve incidents.

### Why

Release confidence depends on detecting emerging failures before users are affected and responding quickly. This work can improve V1 operations and must be ready no later than the V2 rollout.

### Scope

- Define service-health signals for user impact or imminent failure, including stalls, failed recovery, unhealthy queues, account-lifecycle failures, and degraded dependencies.
- Add structured failure reporting and route actionable alerts to the team's incident channel with clear severity, ownership, and diagnostic context that does not expose secrets.
- Define escalation, acknowledgement, recovery, and post-incident workflows, supported by dashboards and runbooks for the agreed production signals.
- Exercise controlled failure scenarios to verify alert reliability and routing, support recovery, and reduce noisy or duplicate notifications.

Completion requires reliable alerts for critical failure modes, dashboards and runbooks that support recovery, and controlled tests confirming correct routing.

Run this work alongside V1 and V2; it may ship before V2.

Related planning: magicblock-labs/dev-kanban#202.