paritytech / paritytech/web3-storage

[Tooling] Add a `storage-cli` tooling crate (on-chain + off-chain ops), starting with `stress-test upload`

Open
#175 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement good first issue
Dominant language
Rust
Stars
12
Forks
3
Avg merge
2d 2h
Merged PRs (30d)
33

Description

Motivation

We've accumulated a bunch of useful commands and snippets for exercising on-chain
and off-chain storage scenarios (registering providers, opening agreements,
uploading data, etc.), currently scattered across examples and ad-hoc scripts.
Consolidate them into one ergonomic binary so contributors and CI can drive both
layers from a single tool.

Proposal

Add a new workspace crate storage-cli under utils/storage-cli, built on
the existing storage-client SDK.

  • New crate utils/storage-cli, added to [workspace].members in the root Cargo.toml.
    (utils/ follows the Polkadot SDK convention of keeping developer tooling under
    substrate/utils/.)
  • clap-based CLI (derive API), one subcommand per operation, with shared global flags:
    • --chain-rpc <URL> (default ws://127.0.0.1:2222)
    • --provider-url <URL> (provider HTTP endpoint; default http://127.0.0.1:3333)
    • --suri <SURI> / --keyfile <FILE> for the signing account
  • Reuse storage-client (StorageUser, discovery, substrate helpers) instead of
    re-implementing chain/HTTP access.

First command: stress-test upload

Simplest case — "upload data to one provider".

Behavior

  1. Resolve target buckets from chain for the (client, provider) pair: read the
    client's buckets (MemberBuckets[client]) and keep those that have an active
    StorageAgreements[bucket][provider] entry (i.e. an agreement between this
    client and the given provider).
  2. If no matching buckets exist -> exit with a clear error. Do not create any
    bucket or agreement.
  3. Upload generated data to each selected bucket via the provider's HTTP endpoint
    (StorageUser::upload).

Params

  • --provider <AccountId> (required) — the provider to target.
  • --max-buckets-to-write <N> (optional) — cap the number of buckets written to;
    default = all matching buckets.
  • --size <BYTES> (optional, suggested) — amount of data to generate per upload,
    with a sensible default.

Acceptance criteria

  • storage-cli crate compiles as a workspace member; cargo run -p storage-cli -- --help
    lists the stress-test subcommand.
  • stress-test upload --provider <id> uploads to all matching buckets;
    --max-buckets-to-write N caps the count.
  • Errors clearly when the client has no agreements with the given provider
    (no silent success, no bucket/agreement creation).
  • Reuses storage-client; no duplicated chain or HTTP logic.
  • --help and a short README document the flow and the required on-chain setup.

Follow-ups (out of scope here)

  • More stress-test modes: replicated upload, multi-provider fan-out, sustained
    load / concurrency.
  • On-chain helper subcommands (register provider, open agreement, submit checkpoint),
    folded in from the existing client/examples/*.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the workspace members in the root Cargo.toml, the existing storage-client SDK, and relevant client/examples/* commands. Create utils/storage-cli with clap-based entry points, then run cargo run -p storage-cli -- --help and exercise stress-test upload against matching agreements. Done means uploads are capped correctly, missing agreements produce a clear error without creating resources, and the README documents setup and flow.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
cli, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.