[Story/Testing] System-wide stress test at pallet limits: agreement turnover, upload load, challenge saturation — with metrics + dashboards

Motivation

We have correctness demos (just demo, just e2e, fs-demo-ci, s3-demo-ci) and #260 will give the provider node performance SLOs — but nothing exercises the system at its configured limits. Every pallet bound (MaxChallengesPerDeadline, MaxBucketsPerMember, MaxPrimaryProviders, …) and every per-block processing path (on_initialize weight reservation, on_finalize challenge slash sweep) is untested at scale. We don't know:

• whether a block with 1,000 maturing challenges (the MaxChallengesPerDeadline bound) actually fits the block weight budget, or what it does to block time;
• what happens when thousands of short-lived agreements expire and nobody sweeps them (lazy end_agreement / claim_expired_agreement by design — #177), i.e. how fast stale state accumulates and what the sweep storm costs;
• whether provider balances/stake/slashes evolve correctly under mass slashing and high-volume checkpoint reward/penalty activity;
• how the provider node behaves while all of the above is happening plus concurrent upload load.

This is the system-level counterpart to #260: #260 measures the provider node in isolation; this issue stresses chain + provider together at the pallet's declared maximums, and makes the whole run observable.

Current state

• No stress tool exists. #175 proposes storage-cli stress-test upload; nothing is built yet. The e2e harness (examples/papi/) is correctness-oriented, single provider, no load dimension.
• The one real per-block loop is the challenge slash sweep: on_finalize in pallet/src/lib.rs does Challenges::drain_prefix(deadline) and slashes every challenge maturing at that block, bounded by MaxChallengesPerDeadline (u16, set to 1,000 in both runtimes); on_initialize pre-reserves the benchmarked weight for it. Saturating this bound requires creating ~1,000 challenges in a single block (all challenges created at block n mature at n + ChallengeTimeout).
• Agreement expiry/settlement is pull-based (by design, #177): expired agreements leave stale rows until end_agreement / claim_expired_agreement is called. A turnover test (many agreements, short durations) will accumulate un-swept state — exactly the residual #177 flags as "optional cleanup"; we should measure how bad it gets.
• Short periods are already achievable on Paseo-based runtimes: runtimes/web3-storage-paseo/src/storage.rs makes ChallengeTimeout, SettlementTimeout, RequestTimeout, DeregisterAnnouncementPeriod storage-backed, and scripts/paseo-lower-timeouts-setstorage.mjs lowers them live via sudo setStorage — built for exactly this ("tuning previewnet timing without redeploying the wasm"). The local runtime uses compile-time constants (48h/24h/6h), so local runs need either a dev preset with short values or the same storage-backed approach.
• Bounds to stretch (local + paseo runtimes, runtimes/*/src/storage.rs): MaxChallengesPerDeadline = 1,000, MaxBucketsPerMember = 1,000, MaxPrimaryProviders = 5, MaxMembers = 100, MaxChunkSize = 256 KiB, MaxNonceAge, DefaultCheckpointInterval = 100 / DefaultCheckpointGrace = 20, MaxDrivesPerUser = 100 (registry pallet).
• Metrics are missing on the off-chain side: the provider node exposes only /health (provider-node/src/api.rs) — no /metrics; the in-memory CheckpointMetrics in client/src/checkpoint.rs is never exported. #214 designs telemetry; this issue is its first real consumer. The chain side is free: polkadot-omni-node / relay nodes already expose standard Substrate Prometheus metrics (block time, block weight ratio, txpool, state size).
• No dashboards / monitoring stack exists (no Grafana, no docker-compose, no exporter). The provider UI (user-interfaces/provider/) is a user-facing console, not an ops dashboard — and on-chain domain state (provider balances, challenge counts, agreement counts) isn't in any node's Prometheus metrics, so it needs a small chain-state exporter.
• CI (.github/workflows/integration-tests.yml) runs zombienet + e2e correctness jobs only; no performance/stress/load job of any kind. Composite actions (start-zombienet, start-e2e-chain, wait-for-parachain, wait-for-provider-health) are reusable for a stress job.
• PreviewNet is a live Paseo-runtime deployment (PPN repo, previewnet.substrate.dev) with the previewnet genesis preset — a realistic long-soak target, already reachable from the tuning script.

Proposed / potential solution

1. Scenarios (driven by the #175 stress tool)

Extend storage-cli stress-test (#175) from upload into a scenario runner with at least:

• stress-test agreements — agreement turnover: N members × M buckets (push toward MaxBucketsPerMember) × very short durations (minutes–hour wall-clock, via lowered timeouts), continuous open → expire → sweep cycles. Measures: extrinsic throughput/failure rates, stale-row accumulation between sweeps, claim_expired_agreement/end_agreement sweep-storm cost, state size growth.
• stress-test challenges — challenge saturation: batch-create challenges targeting the same deadline block, ramping toward MaxChallengesPerDeadline (1,000); run both flavors — provider defends all (proof-generation load on provider) and provider ignores all (mass on_finalize slashing). Measures: block time / block weight at the deadline block, slash correctness, provider free/reserved balance trajectory, NextChallengeIndex cleanup.
• stress-test upload — the #175 command, at concurrency: parallel uploads across all buckets, sized around MaxChunkSize, sustained for the run duration. Measures: provider throughput/latency (ties into #260 SLOs), MMR growth, commit latency.
• stress-test checkpoints — many buckets with the checkpoint coordinator enabled, short CheckpointInterval: checkpoint submission per window across hundreds of buckets, plus deliberate misses + report_missed_checkpoint volume. Measures: reward/penalty accounting at volume, missed-window behavior under load.
• stress-test soak — all of the above concurrently for hours (PreviewNet) — the "everything at once" run.

Each scenario takes --chain-rpc, --provider-url, --scale <profile> and emits machine-readable results (JSON), same convention as #260's harness.

2. Infra (configurable target)

Whatever the target, treat it as a closed sandbox — same spirit as Parity's internal Versi network (or a "zoo" network): disposable, resettable, safe to break at max limits, never user-facing. If PreviewNet gains real users, we should spin up a dedicated stress instance of it instead of soaking the shared one.

• Local / CI: zombienet (reuse start-zombienet / start-e2e-chain composite actions) + inmemory provider; timeouts compressed via a dev genesis preset or setStorage (make the local runtime's timeout params storage-backed like paseo's, or add a stresstest genesis preset with minute-scale values). Nightly/weekly workflow_dispatch-able job with a scale-profile input — not per-PR. Candidate: fold into (or run beside) #260's nightly perf job.
• PreviewNet: long-running soak against the live deployment, timeouts lowered via scripts/paseo-lower-timeouts-setstorage.mjs, restored after the run. Requires coordination with the PPN repo side (funding accounts, provider capacity, cleanup).

3. Metrics collection

• Chain: scrape the omni-node's existing Prometheus endpoint (block time, block weight / on_finalize proportion, txpool size, state size, finality lag).
• Provider node: depends on #214 — /metrics Prometheus endpoint using the Polkadot-SDK metrics crates; export at least upload throughput/latency, commit/proof latency, challenge response time, storage bytes, and surface the existing CheckpointMetrics.
• Domain state exporter (new, small): a PAPI sidecar that polls chain state per block and exports as Prometheus gauges — provider free/reserved balances and stake, active/expired/stale agreement counts, pending challenge counts by deadline, checkpoint reward accruals, slash events, bucket/member counts. This is what makes "see everything (provider balances, …)" possible; nothing existing covers it.
• Persist raw scrapes per run so results are comparable across runs (regression signal, same idea as #260).

4. Dashboards

• Prometheus + Grafana with provisioned dashboards checked into the repo (e.g. monitoring/); for CI runs a docker-compose setup is a possible alternative to spin the stack up ephemerally:
  • Chain health: block time, block weight ratio (watch the challenge-deadline spike), txpool, state size.
  • Economics: per-provider balances/stake/slashes over time, checkpoint rewards vs penalties, challenge deposits.
  • Protocol activity: agreements opened/expired/swept, challenges created/defended/timed-out, checkpoints submitted/missed.
  • Provider node: throughput, latencies, resource usage (from #214 metrics).
• Same stack points at either target (local zombienet or PreviewNet) — only scrape endpoints differ.

5. Pass/fail criteria (make it a test, not just a load generator)

• Block production never stalls; block weight at challenge-deadline blocks stays under budget at MaxChallengesPerDeadline.
• All slashes/rewards/penalties reconcile: sum of balance deltas matches emitted events (run try_state checks from #230 after the run).
• Provider node stays healthy (/health, no OOM/restart) for the full soak.
• No stuck state: after sweeps, no residual expired agreements/challenges beyond the known lazy-cleanup residual (quantified, not hand-waved).

TODO

• Build the scenario runner on top of #175's storage-cli stress-test (agreements / challenges / upload / checkpoints / soak, JSON output)
• Make short timeouts available locally (storage-backed params in the local runtime or a stresstest genesis preset; paseo already covered by paseo-lower-timeouts-setstorage.mjs)
• Provider /metrics endpoint (#214) + surface CheckpointMetrics
• Domain-state Prometheus exporter (provider balances, agreements, challenges, checkpoints)
• monitoring/ stack: Prometheus + Grafana with provisioned dashboards (chain health / economics / protocol activity / provider node)
• CI: nightly/weekly zombienet stress job with scale-profile input (workflow_dispatch), artifacts = JSON results + scrape dumps
• PreviewNet soak runbook (lower timeouts → run → restore; account funding; cleanup; PPN-repo coordination)
• Define pass/fail thresholds; wire try_state (#230) as post-run invariant check
• Baseline report: first full run on current dev at max limits (numbers feed #260 SLOs, #177 hardening decision, #101 DB choice)

Related

• #175 — storage-cli stress-test (the load-generation tool this reuses; prerequisite)
• #260 — provider-node performance SLOs & regression harness (sibling: node-level vs this system-level; shared CI/results conventions)
• #214 — telemetry & metrics design (prerequisite for provider metrics; this issue is its first consumer)
• #177 — on_finalize hardening / lazy processing (this test produces the numbers that decide the "harden challenge sweep" and "expired-agreement residual" questions)
• #230 — try_state invariant checks (post-run verification)
• #233 — relay-block timeouts (changes what "short period" means; stress profiles must be re-derived after it lands)
• #113 — 2s blocks (halves per-block weight budget → rerun challenge-saturation after)
• #157 — challenge cost escalation (mass-challenge economics; griefing-prevention interacts with the challenge scenario)
• #101 / #100 — DB engine / per-bucket isolation (upload/soak numbers are direct input)
• #216 — bandwidth metering (same observability surface)
• #28 — zombienet-sdk rewrite (would simplify the CI stress job)
• #136 — shared chain-waiting utilities (reusable in the runner)