[Provider] Viability & upgrade resilience: version guard, on-chain protocol logic, adaptability to runtime changes

Motivation

A storage provider is meant to be a long-running, high-uptime service. Today the provider binary is tightly coupled to the exact shape of the on-chain runtime — storage layouts, runtime-API signatures, event/extrinsic encodings. That means a runtime upgrade which changes any of those shapes can force a provider rebuild + coordinated redeploy, i.e. downtime, and puts the whole provider fleet on the runtime's release cadence.

Goal: the fewest possible provider-binary upgrades and the least downtime. The provider should adapt to most on-chain changes (new runtime-API versions, evolved storage shapes, tweaked protocol parameters) without a redeploy, and when a redeploy is genuinely required, it should be safe, detectable in advance, and ideally automatic.

This is a topic-scoping issue — we're capturing the problem space and the directions worth investigating, not committing to a design yet.

Problem statement

• The provider decodes on-chain storage and calls runtime APIs directly; changing either on-chain side can break the running binary.
• The typed-bindings direction (#243, storage-subxt) improves safety but tightens the shape coupling: a shape change means regenerating bindings and shipping a new binary. Dynamic decoding is more tolerant but loses type safety. We need a story that gets both.
• There is no explicit version handshake between provider and chain: nothing tells a provider "you are too old / too new for this runtime" before it starts making bad calls.
• A runtime upgrade today has no defined provider-side reaction beyond "operators notice and upgrade."

Directions to investigate (later)

1. Version guard / compatibility negotiation. Define a provider-facing protocol version (distinct from the runtime spec_version). Expose it on-chain; have the provider declare the range it supports and check compatibility on startup and when it observes a runtime upgrade. On mismatch: refuse to start / enter a degraded read-only mode / warn loudly — never silently mis-decode.

2. Stable, versioned provider interface. Insulate the provider from internal runtime refactors behind a curated, versioned RuntimeApi surface (builds on #237) so cosmetic runtime changes don't ripple into the binary.

3. Decoupling from storage shapes. Versioned / encapsulated on-chain structures (#250 Commitment) and forward/backward-compatible encodings so a provider can read both the old and new shape across an upgrade window. Weigh against the typed-bindings work (#243) and storage-iteration hardening (#174).

4. On-chain protocol logic ("mini-runtime"). Put a slice of protocol logic and/or parameters on-chain — from simple version-tagged config up to a small downloadable blob/wasm the provider fetches and hot-applies when its on-chain version changes. Behavior can then evolve without a binary upgrade; the provider just reads → downloads-on-change → applies. Needs careful thought on trust, determinism, and the sandbox/execution model.

5. Automatic / low-downtime binary upgrades. For changes that do require a new binary: a self-update path gated by the version guard, plus graceful/rolling restart and config hot-reload so a provider swaps versions without dropping agreements or challenge responsiveness.

6. Chain-connection resilience across upgrades. How the provider stays connected and reacts to a runtime upgrade at runtime — ties into the light-client-vs-RPC investigation (#222) and event-driven coordinators (#227).

7. Observability of version drift. Surface provider↔chain version compatibility as a first-class metric/alert so incompatibility is caught before it causes missed checkpoints or challenges (ties into telemetry #214).

Out of scope (for now)

Concrete design, the execution/sandbox model for any on-chain logic, and the exact versioning scheme — all deferred to follow-up investigation.

Relevant / related issues

• #237 — Expose interfaces via a separate RuntimeApi (stable versioned surface)
• #243 — Migration: subxt::dynamic → typed storage_subxt bindings (shape-coupling tradeoff)
• #250 — Encapsulate the Commitment structure (versionable storage shapes)
• #174 — Harden subxt storage iteration / key parsing (robust decoding)
• #222 — Provider chain connection: smoldot light client vs RPC + event-driven coordinators
• #227 — Event-driven coordinators (reacting to on-chain changes)
• #214 — Design telemetry and metrics for on-chain and off-chain components
• #178 — Repo structure / storage-subxt home (where metadata bindings live)
• #44 — Production readiness: Web3 principles + Polkadot SDK standards
• #48 — Deploy MVP to Paseo (real upgrade cadence to design against)

No files or tests are named. Start by reading the linked RuntimeApi (#237), typed storage_subxt bindings (#243), Commitment encoding (#250), and connection-resilience (#222) issues, then map how the provider currently calls runtime APIs and decodes storage. Done means producing a scoped follow-up investigation or design; concrete implementation is explicitly deferred.