paritytech / paritytech/web3-storage

[Provider] Performance SLOs and a regression harness for the provider node

Open
#260 0 comments 0 reactions 1 assignee View on GitHub

@dr333ws is already working on this.

Since Jul 2, 2026.

enhancement
Dominant language
Rust
Stars
12
Forks
3
Avg merge
2d 2h
Merged PRs (30d)
33

Description

Motivation

"Performant" is currently unmeasured: the provider node has no benchmarks, no performance targets, and no CI signal that a change made uploads, commits, or proof generation slower. Meanwhile we already know the hot paths have structural problems (the MMR is fully rebuilt on every commit and every proof — #100), and the fix work needs numbers to be evaluated against. Without a harness, every performance claim — including "this refactor didn't regress challenge response" — is guesswork, and challenge response latency is stake-at-risk, not cosmetics.

Current state

  • No benches/ directory and no criterion dependency anywhere in the workspace; the only performance-adjacent tooling is FRAME weight benchmarking for extrinsics (on-chain costs, not provider throughput).
  • No defined targets: nothing states what upload throughput, commit latency, or proof-generation p99 a provider must sustain, so there is nothing to test against (also blocks #42's "provider min requirements" and #48's disk sizing).
  • #100/#101 propose one-off measurements to pick a storage architecture/engine — valuable, but one-off; nothing institutionalizes the numbers or catches regressions after the decision.
  • #175's storage-cli stress-test upload is the natural load-generation vehicle but isn't built yet.

Proposed / potential solution

  • Define SLOs first (a short doc, then encode as thresholds): e.g. sustained upload MB/s per stream, commit latency vs bucket size, MMR/chunk-proof generation p99 vs leaf count, challenge response end-to-end budget (poll → proof → tx submitted), startup/recovery time vs stored bytes.
  • Micro-benches: criterion benches for the hot paths (mmr append/proof, storage/disk commit + proof generation, auth/signing) parameterized by leaf count (1k/10k/100k) so the O(n)-rebuild problem of #100 becomes a visible curve, and its fix a visible win.
  • Macro harness: build storage-cli stress-test (#175) into a repeatable scenario runner (N buckets × M uploads × concurrent reads + a challenge in flight) against a real provider + zombienet chain; emit machine-readable results.
  • CI wiring: a periodic (nightly/weekly, not per-PR) job that runs both layers on a fixed-spec runner, stores results, and alerts on threshold breach or significant delta; per-PR runs opt-in via the /cmd bot.

TODO

  • Triage and prepare a partial design of what to implement (which SLOs, benches and harness pieces come first)
  • SLO doc with initial target numbers (inputs for #42 hardware requirements and #48 Paseo sizing)
  • criterion micro-benches for mmr/commit/proof paths, parameterized by leaf count
  • storage-cli stress-test scenario runner with machine-readable output (#175)
  • Nightly perf CI job + regression alerting; /cmd hook for on-demand PR runs
  • Baseline report on current main (pre-#100 numbers, so the improvement is documented)

Related

  • #100 / #101 — the storage-architecture decisions this harness must measure (before/after)
  • #175 — storage-cli stress-test command (load generator)
  • #214 — telemetry; the same metrics should be exported by production providers
  • #42 — provider hardware requirements / cost-benefit needs these numbers
  • #113 — 2s blocks change the challenge/checkpoint timing budget

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.