[Provider] Graceful shutdown & ordered startup (signal handling, drain, crash-safe persistence)

Subtask of #254: the restart mechanics needed for low-downtime provider upgrades. This covers clean shutdown and startup only; self-update and config hot-reload stay in #254.

Motivation

A provider restart is unavoidable (binary upgrades, host maintenance, crashes) — and the protocol tolerates it, but on two very different clocks: challenges leave a 48h response window, while provider-initiated checkpoints run on ~10-minute windows with only a ~2-minute grace (DefaultCheckpointInterval = 100 blocks, DefaultCheckpointGrace = 20) — a restart that overruns the grace while the provider is checkpoint leader for a bucket can already be hit with report_missed_checkpoint → CheckpointMissPenalty. So downtime must be clean and bounded against the checkpoint grace, not the challenge window: today the provider cannot be stopped without aborting in-flight work, and it cannot tell an orchestrator (systemd, k8s, a future self-update path in #254) when it is actually ready or actually done.

Every low-downtime upgrade story in #254 bottoms out on this primitive: stop accepting work → finish/flush what's in flight → exit with intent → come back up in a known-good order. It doesn't exist yet.

Current state (evidence)

• No signal handling at all. The HTTP server is started as plain axum::serve(listener, app).await (provider-node/src/command.rs:130-134) with no .with_graceful_shutdown(...); there is no tokio::signal / SIGTERM / SIGINT handler anywhere in provider-node. A kill drops in-flight uploads, commits, and challenge responses mid-flight.
• Coordinator lifecycle is manual and unwired. ChainStateCoordinator / CheckpointCoordinator / ReplicaSyncCoordinator / ChallengeResponder each have a *Handle::stop() that just aborts the task (e.g. chain_state_coordinator.rs:427-435), but nothing calls them on shutdown — tasks are dropped abruptly when the runtime exits.
• Persistence is clean-shutdown-only. RocksDB relies on flush-on-drop ("normal shutdown" comment at storage/disk.rs:573) — not crash-safe; the fs/s3 index managers persist JSON snapshots that can be torn on abrupt exit. The nonce high-water mark (DiskNonceStore, recovered via max(persisted, hsn+1) in chain_state_coordinator.rs:316-337) is the one piece that is restart-aware by design.
• Startup has no ordering or readiness gate. Storage open, chain connect, multiaddr sync, coordinator spawn, and HTTP bind happen without a readiness distinction — /health reports liveness before the chain state is reconciled, so an orchestrator can route traffic (or a supervisor can consider the upgrade "done") while the provider is still bootstrapping.

Proposed scope

1. Shutdown

• Install SIGTERM/SIGINT handlers; wire them into axum::serve(...).with_graceful_shutdown(...).
• Define and implement a drain sequence with a configurable overall deadline (e.g. --shutdown-grace-secs, default ~30s — the full stop→start cycle should fit comfortably inside DefaultCheckpointGrace to avoid missed-checkpoint penalties):
  1. flip to draining: reject new mutating requests (uploads, commits, negotiate) with 503 + Retry-After; keep serving reads;
  2. let in-flight requests complete (bounded by the deadline);
  3. stop coordinators in dependency order — challenge responder last, and let it finish an already-started challenge response (that's stake at risk);
  4. flush storage: RocksDB flush + WAL sync, fs/s3 index snapshots written atomically (write-temp + rename), persist nonce HWM;
  5. exit 0 on a clean drain, distinct non-zero code on deadline-forced exit.
• Make index snapshot writes atomic regardless of shutdown (crash-safety, not just SIGTERM-safety).

2. Startup

• Ordered bootstrap: open storage (+ cheap integrity check of index vs DB) → connect chain → reconcile provider state (constants, ProviderInfo, hsn/nonce) → sync multiaddr → start coordinators → then report ready.
• Split readiness from liveness: keep /health as liveness; add /ready that only turns true after chain reconciliation completes. Document both for operators.
• Log a single structured "startup summary" line (chain, spec_version, provider account, buckets served, recovery actions taken) — feeds #214.

3. Integration test

• Start provider → begin an upload + a pending challenge → SIGTERM → assert clean drain, restart, assert challenge still answered and no data loss.

Related

• #254 — parent (low-downtime binary upgrades)
• #222 — chain connection / reconnect behaviour on the other side of a restart
• #214 — readiness/liveness + startup metrics
• #178 — the drain logic should land in whatever crate split owns the coordinators
• #149 — audit item P11 ("no graceful shutdown handler in provider node") is superseded by this issue

Start with provider-node/src/command.rs:130-134 and the coordinator stop handles, then inspect storage/disk.rs:573 and the nonce recovery logic in chain_state_coordinator.rs:316-337. Trace how HTTP serving, coordinator tasks, and index persistence currently start and stop. Done means bounded signal-driven draining, ordered bootstrap with separate liveness/readiness, crash-safe persistence, and the described restart integration test passing.