[Investigation] Provider chain connection: smoldot light client vs RPC node + event-driven coordinators

Relates to: #207
Relates to: https://github.com/paritytech/web3-storage/pull/212#issuecomment-4736621910
Relates to: https://github.com/paritytech/web3-storage/pull/105/changes#r3387123030

Goal

Let the provider node choose how it talks to the chain — either an external RPC node (current behaviour) or an embedded smoldot light client — and make smoldot (via subxt light client) the default, so a provider needs no operated RPC infrastructure to run.

This mirrors the bridges investigation in paritytech/parity-bridges-common#3270 (relayers on smoldot instead of RPC nodes); the same "no infra" motivation applies to storage providers.

Investigated 2026-07-02 (full chain-surface audit of provider-node + client, plus subxt/smoldot upstream recon). Verdict: green light — no upstream blocker. Unlike the bridges case (which is gated on smoldot exposing GRANDPA justifications, paritytech/smoldot#3288), the provider needs no justifications and no storage proofs, and its access patterns are already light-client-shaped. The switch really is a transport swap; the open questions from the original Scope are answered below.

Current state

• The provider connects with subxt::OnlineClient::<PolkadotConfig>::from_url(chain_rpc) — a plain ws:///wss:// URL to an external node. Five call sites total: provider-node/src/subxt_client.rs, provider-node/src/auth.rs, and in the client crate block_subscription.rs, event_subscription.rs, substrate.rs.
• subxt is pinned at 0.44.3, default features → legacy JSON-RPC backend over jsonrpsee-WS. No backend selection, no light-client feature enabled.
• All background coordinators (checkpoint, replica-sync, challenge-responder) periodically poll the chain on fixed intervals (6s / 12s); chain_state_coordinator already follows subscribe_finalized() and reacts to events.

Why the code already fits a light client (audit results)

• Every storage read is .at_latest() — zero historical .at(hash) reads anywhere, so the light client's pinned-recent-blocks-only limitation never bites. No archive node needed.
• No runtime-API calls, no raw/legacy RPC calls (state_*/chain_*/system_*) in application code. The only implicit node dependency is subxt's automatic account-nonce fetch on tx submission, which a light-client backend also satisfies.
• Reconnect/bootstrap is already the light-client-friendly pattern: chain_state_coordinator::connect_and_follow re-reads current state wholesale on (re)connect instead of back-scanning missed events — no deep event scans exist.
• All 4 extrinsics (update_provider_multiaddr, provider_checkpoint, confirm_replica_sync, respond_to_challenge) use sign_and_submit_then_watch_default + wait_for_finalized_success() — and smoldot's chainHead backend delivers real finalized tx events (it follows finality by construction).
• No libp2p stack in the binary (storage transfer is HTTP via axum/reqwest; on-chain multiaddrs are just HTTP endpoint advertisements) — embedded smoldot would be the first p2p stack here, no conflict or duplication.
• Chain topology: this is a Cumulus/Aura parachain (para 4000, Westend/Paseo relay). smoldot syncs parachains by warp-syncing the relay and deriving para finality from it — no para-side justifications involved. Embedded Rust smoldot-light dials ordinary TCP bootnodes directly (the WebSocket requirement is browser/WASM-only).

Scope (updated)

1. Configurable transport — smoldot (default) or rpc. Confirmed drop-in shape with subxt ≥ 0.50:

   let (lightclient, relay_rpc) = LightClient::relay_chain(RELAY_SPEC)?;
   let para_rpc = lightclient.parachain(PARA_SPEC)?;
   let api = OnlineClient::<PolkadotConfig>::from_rpc_client(para_rpc).await?;
   

   The existing OnlineClient call sites don't change — only construction does (the 5 from_url sites above).

   Prerequisite: bump subxt 0.44.3 → 0.50.1. This matters beyond API convenience: subxt 0.44.x pulls smoldot-light 0.17.2 from the abandoned pre-revival lineage, while subxt 0.50.1 moved to smoldot-light ^1.1.0 (the revived paritytech/smoldot line with the 2026 fixes: elastic scaling paritytech/smoldot#3141, warp-sync retry/ban fixes, Kademlia discovery). 0.50 also adds the CombinedBackend, which auto-routes to the chainHead backend against smoldot. Note the light-client feature is still officially experimental (paritytech/subxt#1811).

   Chain-spec bundling (was an open question): ship relay + para specs. For Paseo, paseo-network/paseo-chain-specs provides light-client-optimized *.raw.smol.json variants; our own chain-specs/ needs the para spec to carry reachable bootnodes serving the light request-response protocols (on by default in substrate/cumulus nodes). Keep lightSyncState checkpoints not-too-fresh (paritytech/smoldot#3271).

2. Coordinator polling to be revisited — #81 is not just an optimization, it's what makes the smoldot default robust. The audit found the hot spot: full storage-map iterations — Challenges every 6s, StorageAgreements every 12s, Providers in discovery. Storage iteration over the light client is slow and currently buggy (paritytech/subxt#1743 — duplicate/hanging iterator items, still open; perf: paritytech/subxt#1911). Point reads and per-block event reactions (what chain_state_coordinator already does) are the supported happy path on the chainHead backend. So the event-driven redesign should land with (or before) the smoldot default; until then, prefer targeted point reads over map scans.

Transport alternatives — why subxt, and which escape hatches to keep

Short version: keep subxt for the API layer (the provider is already subxt-native — dynamic storage, signer, OnlineClient everywhere), and note that smoldot is the only maintained light-client implementation of GRANDPA warp-sync in any language — so every "alternative" is a different way of wiring smoldot in, not a different light client. Three viable attachments:

1. subxt's built-in LightClient (subxt-lightclient) — the default choice. Drop-in via from_rpc_client, maintained by the same team, tracks smoldot-light 1.x since 0.50.1. Downside: we inherit subxt's smoldot pin and experimental status, and the silent-hang failure mode (paritytech/subxt#1536) lives inside our process — the watchdog is on us.
2. Direct smoldot-light behind a custom subxt transport — the escape hatch. subxt 0.50's RpcClientT trait is just two methods (request_raw, subscribe_raw; rpcs/src/client/rpc_client_t.rs), and subxt's own light-client wiring is exactly one such impl (lightclient_impl.rs). A custom impl against current smoldot-light 1.x is a few hundred lines and buys: our own smoldot version (upstream fixes ship monthly, subxt bumps lag), platform tuning, and first-class watchdog/restart-in-place hooks. Move here only if option 1's version coupling or hang behavior bites in practice.
3. smoldot as a co-located sidecar process exposing WS. A tiny separate binary (smoldot-light + WS server, or the smoldot npm package under Node) that the provider reaches via the existing from_url — zero provider-code change, and it converts the silent-hang class into a supervisable process crash, restartable independently of the provider. Still "no infra" (local process, not an operated node), but an extra artifact to ship — fights the single-binary appeal.

Not real alternatives: polkadot-api (PAPI) + smoldot is TypeScript-only (a rewrite, not a transport swap); substrate-connect is browser-only; substrate-api-client has no light-client backend; hand-rolled jsonrpsee (the bridges relayer's approach) is strictly more work than subxt here; Substrate's full-node "light mode" was removed years ago, and smoldot's experimental full-node binary is still a node to operate.

Design consequence: construct the chain client from an injected RpcClient rather than hardcoding LightClient — then options 2 and 3 stay configuration-level fallbacks instead of refactors, and the smoldot/rpc switch from Scope item 1 falls out of the same seam.

Known operational issues to design around (from upstream, 2026)

• Silent hangs on smoldot panics: a smoldot background-thread panic leaves subxt's finalized stream blocked forever with no error (paritytech/subxt#1536). A finality-stream watchdog + client restart is mandatory. Related open parachain-sync panics observed on Paseo networks: paritytech/smoldot#3286 (relay-block double-unpin), paritytech/smoldot#3159.
• Startup warm-up: tx submission right after start fails until enough peers are connected (~20s; paritytech/subxt#1524) — coordinators should gate their first submission on sync-ready.
• chainHead stop events can disrupt in-flight tx watches and drop pinned blocks (paritytech/subxt#1769); handle by resubscribing (missed-finalized-blocks handling improved by paritytech/smoldot#3282).
• Encouraging: Parity is actively dogfooding exactly this shape (embedded smoldot against Paseo parachains) in 2026, so these rough edges have live upstream attention.

Notes

This is an investigation/tracking issue — not committing to an implementation yet. The investigation above settles feasibility (doable, no upstream dependency); remaining design work is the subxt 0.50 bump, transport config + chain-spec packaging, the watchdog, and sequencing with the event-driven coordinator redesign (#81).

Edited 2026-07-02: added full chain-surface audit results, subxt/smoldot version findings (0.44 → 0.50.1 prerequisite), chain-spec/bootnode answers, upstream issue references, and the transport-alternatives analysis. Original open questions in Scope are now answered inline.