paritytech / paritytech/web3-storage

Repo structure: split provider-node, break the storage-client dependency, tidy crate layout

Open
#178 4 comments 0 reactions 1 assignee View on GitHub

@danielbui12 is already working on this.

Since Aug 24, 2026.

Dominant language
Rust
Stars
12
Forks
3
Avg merge
2d 2h
Merged PRs (30d)
33

Description

Why

  • provider-node depends on storage-client — an inverted edge (server pulling in the client SDK).
  • provider-node is one ~8.4k-LOC crate mixing storage, HTTP, and on-chain coordinators.
  • Crate layout is inconsistent (runtime/ vs runtimes/, dir names != crate names, flat vs nested).

Also sets up a clean home for utils/storage-cli (#175).

1. Break provider-node -> storage-client

What it actually pulls from storage-client is two unrelated things:

  • Pure wire types (no I/O): negotiate agreement::{NegotiateRequest, AgreementTermsOf, SignedTerms} + sign_terms + discovery::ProviderInfo, used server-side in the /negotiate handler.
  • A second chain client: chain_state_coordinator.rs uses storage_client::ProviderClient to talk to the parachain over WS — get_provider_info, fetch_replay_hsn, fetch_request_timeout, plus a finalized-block subscription parsed with StorageProviderEventParser + substrate::PALLET_NAME.

⚠️ The provider already has its own chain client. subxt_client.rs (SubxtChainClient, ~917 LOC) talks to the same parachain directly via subxt. So today there are two parallel chain-access paths in one node — the node's own SubxtChainClient and the reads/subscription routed through storage_client::ProviderClient. Step 1 isn't just deleting a types dependency; it collapses those two clients into one.

  • Extract storage-negotiation primitive crate (wire types + MultiSignature serde + sign_terms); AgreementTerms etc. already live in primitives.
  • Move the chain reads + event subscription (get_provider_info, fetch_replay_hsn, fetch_request_timeout, finalized-block stream) onto the node's own SubxtChainClient (post-#159), retiring the ProviderClient path.
  • Drop storage-client from provider-node.

2. Split provider-node -> provider/{storage,rpc,node}

Crate Modules subxt?
provider/storage storage/*, fs_index, s3_index, mmr no
provider/rpc (HTTP API) api, fs_api, s3_api, auth, negotiate, error, types no
provider/node (bin) main, command, cli, lib, subxt_client, coordinators yes

Direction: node -> rpc -> storage -> primitives; only node touches the chain. rpc gets chain data via traits it defines, with subxt-backed impls supplied by node (extends #145).

3. Crate layout

Changed (Rust crates that move / get created)
pallets/                   <- consolidates scattered pallets
  storage-provider/        was pallet/                                   (pallet-storage-provider)
  drive-registry/          was storage-interfaces/file-system/pallet-registry
  s3-registry/             was storage-interfaces/s3/pallet-s3-registry
primitives/
  storage/                 was primitives/                               (storage-primitives)
  file-system/             was storage-interfaces/file-system/primitives
  s3/                      was storage-interfaces/s3/primitives
  negotiation/             NEW — wire types, MultiSignature serde, sign_terms (step 1)
provider/                  <- was the single provider-node/ crate
  storage/                 storage/*, fs_index, s3_index, mmr
  rpc/                     api, fs_api, s3_api, auth(core), negotiate, error, types
  node/                    main, command, cli, lib, subxt_client, coordinators  (the bin)
runtimes/
  parachain/               was runtimes/web3-storage-local               (storage-parachain-runtime)
  paseo/                   was runtimes/web3-storage-paseo
clients/                   <- was client/ + storage-interfaces/*/client
  storage/                 was client/                                   (storage-client)
  file-system/             was storage-interfaces/file-system/client
  s3/                      was storage-interfaces/s3/client
utils/
  storage-cli/             NEW (#175)

Net effect: pallet/, primitives/, client/, provider-node/, and the whole
storage-interfaces/ tree are dissolved into the role-based groups above;
runtimes/ subdirs are renamed. Align dir names to crate names while moving
(today every dir mismatches its crate, e.g. client/ -> storage-client).

Unchanged (stay where they are)
precompiles/               drive-registry, s3-registry, storage-provider
packages/                  JS/TS SDK: core, layer0, layer1, papi, sdk (the old `sdk/ ???` slot)
user-interfaces/           JS/TS UIs: landing, drive-ui, provider, s3-ui, console-ui, shared
zombienet/                 zombienet.toml + local network config
chain-specs/  docs/  examples/  templates/  scripts/  .bin/  .config/  .github/

Their contents don't move, but the mechanical reorg PR must still fix path
references
inside scripts/*.sh, zombienet.toml, the justfile, CI
workflows under .github/, and docs that cite moved crate paths.

Note: runtime/ vs runtimes/ (in "Why") is already resolved — only runtimes/ exists today.

Plan (each its own PR; reorg last)

  • Break the coupling (step 1)
  • Split provider-node — storage first, then rpc, leaving node as the bin
  • Directory reorg — one mechanical git mv PR fixing Cargo paths + CI + scripts together

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.