paritytech / paritytech/web3-storage

[Tooling] `storage-cli stress-test upload`: let the CLI prepare its own buckets, outside the measured window

Open
#366 3 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Rust
Stars
12
Forks
3
Avg merge
2d 2h
Merged PRs (30d)
33

Description

Motivation

stress-test upload (added in #220) can only write to buckets that already exist. discover_target_buckets in crates/utils/storage-cli/src/commands/stress_test.rs reads MemberBuckets[account], keeps the buckets with a StorageAgreements[bucket][provider] entry, and bails out when nothing matches. Every caller therefore has to arrange the on-chain state out of band before the CLI is useful: CI seeds it with a separate storage-client --example complete_workflow step, and a human running the tool locally has to do the same by hand.

That makes the tool awkward to point at a fresh chain, and it makes the shape of a run depend on whatever bucket topology happened to be left behind by a previous step. Running against N buckets instead of 1 requires N manual seeding invocations.

The CLI should be able to create the buckets and agreements it needs. The important constraint is that this preparation is setup, not load: it must not appear anywhere in the reported throughput or latency numbers.

Current behavior

1. resolve signer from --suri/--keyfile
2. discover buckets on-chain (read-only), error if none
3. start the clock, run users x uploads-per-user
4. summarize

Bucket and agreement creation is explicitly out of scope of the command today, and the README documents it as a required manual precondition.

Proposal

Add a preparation phase to stress-test upload that provisions the target buckets itself, and keep it strictly outside the measured window.

New flags
Flag Default Description
--prepare-buckets <N> 0 (off) Create N fresh buckets with a primary agreement to --provider before the run, and use them as the upload targets.
--prepare-max-bytes <BYTES> derived from the load max_bytes negotiated per prepared bucket. Default should cover users * uploads-per-user * payload-size spread across the prepared buckets, so a run cannot fail on an undersized agreement.
--prepare-duration <BLOCKS> sensible default Agreement duration for prepared buckets.
--prepare-price-per-byte <P> sensible default Price offered when negotiating terms.
--bucket-source <MODE> discover discover (today's behavior), create (only use freshly prepared buckets), auto (use discovered buckets, prepare more only if fewer than requested exist).

--prepare-buckets and --max-buckets-to-write should compose predictably, and the command should still fail fast with a clear message when the resulting target set is empty.

Preparation mechanics

Reuse the SDK path the complete_workflow example already uses, so no chain or HTTP logic is duplicated in the CLI:

  1. ProviderClient::negotiate_terms(provider_url, NegotiateRequest { owner, max_bytes, duration, price_per_byte, .. }) to get SignedTerms.
  2. AdminClient::establish_storage_agreement(provider, signed_terms) to redeem them, which opens the bucket and returns its BucketId.
  3. Repeat --prepare-buckets times, collecting the ids into the same Vec<BucketId> the round-robin assignment already consumes.
Preparation must be excluded from the performance check

This is the part that matters most for the numbers to stay meaningful:

  • The measurement clock (let started = Instant::now() in upload) starts after preparation completes, so agreement negotiation and extrinsic finalization never inflate elapsed_secs or deflate throughput.
  • Preparation produces no OpOutcome and is never passed to summarize. Latency percentiles stay derived from upload operations only.
  • Preparation failures are setup failures: the command exits non-zero with context, rather than folding the error into failed in the summary. Per-upload failures keep their current, more forgiving semantics.
  • Preparation progress goes to stderr, keeping stdout parseable for --output json.
  • Optional but useful: report preparation separately (for example a preparation object next to, not inside, the operation summaries) with bucket count, ids, and wall-clock time, so an operator can see setup cost without it contaminating the load metrics.
Follow-on cleanup

With this in place, the Seed bucket + agreement (//Bob → //Alice provider) step in the utils-integration-tests job can be dropped and the CLI invocation given --prepare-buckets, so the job exercises the tool end to end from a clean chain.

Acceptance criteria

  • stress-test upload --prepare-buckets N succeeds against a chain and provider where the signing account owns no buckets at all.
  • Prepared buckets receive uploads via the existing round-robin assignment, identically to discovered ones.
  • The reported elapsed_secs, throughput_*, and latency_* values cover only the upload phase. A run with --prepare-buckets 5 and a run against 5 pre-existing buckets report comparable throughput for the same load.
  • A preparation failure (provider refuses terms, agreement extrinsic fails, chain unreachable) exits non-zero and prints the cause, with no metrics summary claiming a successful run.
  • --output json on stdout stays valid JSON with preparation enabled.
  • Default --prepare-max-bytes is large enough that the configured load cannot exceed the agreement, or the command errors up front if it would.
  • crates/utils/storage-cli/README.md documents the new flags and drops the manual complete_workflow precondition from the required-setup section.
  • Unit tests cover flag validation and the --bucket-source selection logic; the CI job runs the CLI without a separate seeding step.

Out of scope

  • Benchmarking bucket preparation itself. Bucket and agreement creation is setup, and it is explicitly NOT part of the performance benchmark. None of its cost - term negotiation, extrinsic submission, finalization wait - may be counted in the reported numbers. The measured window covers uploads and nothing else, so results from a run that prepared its own buckets stay directly comparable to a run against pre-existing ones.
  • If we ever do want throughput figures for agreement establishment, that is a separate subcommand with its own Operation and its own summary, never folded into the upload metrics.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in crates/utils/storage-cli/src/commands/stress_test.rs, tracing upload, discover_target_buckets, and the existing complete_workflow SDK path. Review crates/utils/storage-cli/README.md and the utils-integration-tests job for setup and invocation changes. Done means preparation supports the new flags, remains outside upload metrics, preserves valid JSON output, and has validation and selection tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
cli
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.