Scalable Web3 Storage: Gap Analysis

tags: `web3-storage` `substrate` `polkadot`

Overview

This document compares the design specification against the current implementation to identify completed features, in-progress work, and remaining gaps.

The specific design-doc↔code drifts are also marked inline in the codebase (greppable DRIFT-NNN markers) in #376.

Refreshed: September 2026 (re-verified against origin/dev). Two large deltas landed since the July 2026 revision (which was pinned to 9edd13f): provider-initiated checkpoints + the checkpoint reward pool were fully removed (pallet dropped 32 → 27 extrinsics), and the provider node was split into crates/providers/* workspace crates (auth, chain, challenge, coordinator, negotiation, storage). Both are reflected inline below. The March 2026 revision remains superseded by the delta box.

What changed since the March 2026 revision

Area	Then	Now
Agreement negotiation	On-chain request/accept/reject extrinsics	Moved off-chain (#207). On-chain keeps only mutual commitment via `establish_storage_agreement` / `establish_replica_agreement`. Pallet dropped from 35 → 32 → 27 extrinsics (the last −5 from removing provider-initiated checkpoints, below)
Provider-initiated checkpoints	✅ (July): `provider_checkpoint` + reward pool + missed-checkpoint penalties	❌ Removed (#306/#311). Only client-submitted `checkpoint` / `extend_checkpoint` remain
Provider node layout	Flat modules under `provider-node/src/`	*Split into `crates/providers/`** crates (auth, chain, challenge, coordinator, negotiation, storage); `provider-node` is now the thin binary
Challenge cost-sharing sliding scale	❌ "Not implemented" (flagged 🔴 Critical)	✅ Implemented (#125 Challenge flow overhaul): 90/80/70/60/50% split by response time
Burn option	❌ Not implemented	✅ Implemented — `EndAction::Burn { burn_percent }` handled in `end_agreement`
SDK story	Rust client SDK (12 modules) as primary	TypeScript SDK is now primary (#160): `packages/{core,layer0,layer1,papi,sdk}`. Rust `client/` still present
Smart contracts	Not covered	3 `pallet_revive` precompiles: storage-provider, drive-registry, s3-registry
User interfaces	3 (Console, Provider, Drive)	drive-ui, provider (dashboard), s3-ui, photos, landing (static). `console-ui` is now legacy/dead
Runtimes	1 (local)	2: web3-storage-local + web3-storage-paseo; both add `Utility`, `Revive`, `S3Registry`, `WeightReclaim`
CI/CD	3 workflows	11 workflows incl. try-runtime migration check, security (zizmor + gitleaks), release, UI checks, command bot
Tests	~201	~650+
Toolchain	Rust 1.88 / stable2512-2	Rust 1.93.0 / polkadot-stable2603

Legend

Symbol	Meaning
✅	Implemented
⚠️	Partial/Simplified
❌	Not Implemented

1. On-Chain (Pallet `pallet-storage-provider`)

Feature	Status	Notes
Buckets	✅	Creation (via agreement), membership, min_providers
Atomic Bucket+Agreement	✅	`establish_storage_agreement` creates bucket + agreement in one shot
Off-Chain Agreement Negotiation	✅	#207 — negotiation happens off-chain; only mutual commitment lands on-chain
Bucket Freezing	✅	Append-only mode via `frozen_start_seq`
Bucket Permissions (RBAC)	✅	Admin/Writer/Reader via `set_member`, `remove_member`
Member Bucket Index	✅	`MemberBuckets` storage map
Provider Registry	✅	Stake, multiaddr, stats
Provider Capacity	✅	`max_capacity` with `MinStakePerByte` validation
Two-Phase Deregistration	✅	`deregister_provider` → `complete_deregister` / `cancel_deregister`
Storage Agreements	✅	Primary + Replica
Agreement Extensions / Top-Up	✅	`extend_agreement`, `top_up_agreement`
Agreement Ownership Transfer	✅	`AgreementOwnershipTransferred` event
Extensions Blocking	✅	`set_extensions_blocked`
Replica Agreements	✅	Per-sync payments drawn from `sync_balance`
Checkpoints	✅	Client-submitted `checkpoint` / `extend_checkpoint`; MMR root with multi-provider signature verification
Provider-Initiated Checkpoints	❌	Removed (#306/#311) — `provider_checkpoint`, `configure_checkpoint_window`, `report_missed_checkpoint`, `claim_checkpoint_rewards`, `fund_checkpoint_pool` all gone
Checkpoint Pool & Rewards	❌	Removed (#306/#311)
Missed Checkpoint Penalties	❌	Removed (#306/#311)
Challenge System	✅	3 variants: checkpoint, off-chain, replica
Challenge Cost-Sharing Sliding Scale	✅ NEW	Response-time split: block 1 → 90/10, ≤5 → 80/20, ≤24 → 70/30, ≤95 → 60/40, 96+ → 50/50
Batched Challenge Slashing	✅ NEW	`PendingChallenges` swept on `on_finalize`; `ChallengerStats` tracked
Challenge Settlement	✅	Grace period after expiry for final claims
Burn Option	✅ NEW	`EndAction::Burn { burn_percent }` burns a % on agreement end
Replica Sync Confirmation	✅	`confirm_replica_sync`, `top_up_replica_sync_balance`
Historical Roots	✅	6 prime-based positions for late replica sync
Weights/Benchmarks	✅	FRAME benchmarking for all extrinsics
Proof-of-DOT	❌	Phase 3 — not started

Main Pallet Extrinsics (27, call-index count)

Category	Extrinsics
Provider mgmt	`register_provider`, `add_stake`, `deregister_provider`, `complete_deregister`, `cancel_deregister`, `update_provider_settings`, `update_provider_multiaddr`, `set_extensions_blocked`
Bucket mgmt	`set_min_providers`, `freeze_bucket`, `set_member`, `remove_member`, `remove_slashed`
Agreements	`establish_storage_agreement`, `establish_replica_agreement`, `end_agreement`, `claim_expired_agreement`, `top_up_agreement`, `extend_agreement`
Checkpoints	`checkpoint`, `extend_checkpoint`
Challenges	`challenge_checkpoint`, `challenge_offchain`, `challenge_replica`, `respond_to_challenge`
Replica sync	`confirm_replica_sync`, `top_up_replica_sync_balance`

Removed (agreement negotiation moved off-chain, #207): create_bucket, create_bucket_with_storage, request_agreement, request_primary_agreement, accept_agreement, reject_agreement, withdraw_agreement_request.
Removed (provider-initiated checkpoints, #306/#311): provider_checkpoint, configure_checkpoint_window, report_missed_checkpoint, claim_checkpoint_rewards, fund_checkpoint_pool.
Added: complete_deregister, cancel_deregister, establish_storage_agreement, establish_replica_agreement.

Query/state access is now served via a runtime API (query_provider_info, query_bucket_info, query_available_providers, query_find_matching_providers, query_challenges_at, …) rather than extrinsics.

Pallet Events (31)

Provider (8): ProviderRegistered, ProviderDeregistered, DeregisterAnnounced, DeregisterCancelled, ProviderStakeAdded, ProviderSettingsUpdated, ProviderMultiaddrUpdated, ExtensionsBlocked
Bucket (10): BucketCreated, BucketFrozen, BucketDeleted, MemberSet, MemberRemoved, BucketCheckpointed, ProviderAddedToBucket, PrimaryProviderRemoved, PrimaryAgreementEndedEarly, SlashedProviderRemoved
Agreement (8): AgreementAccepted, AgreementToppedUp, AgreementExtended, AgreementOwnershipTransferred, AgreementEnded, AgreementExpiredClaimed, StorageAgreementEstablished, ReplicaAgreementEstablished
Replica (2): ReplicaSynced, ReplicaSyncBalanceToppedUp
Challenge (3): ChallengeCreated, ChallengeDefended, ChallengeSlashed

Checkpoint events (ProviderCheckpointSubmitted, CheckpointConfigUpdated, CheckpointMissPenalized, CheckpointRewardClaimed, CheckpointPoolFunded) removed with provider-initiated checkpoints (#306/#311).

Pallet Storage Items (12)

Storage	Purpose
`Providers`	Provider registry (info, settings, stats)
`ProviderReplayStates`	Anti-replay state per provider
`NextBucketId`	Bucket ID counter
`Buckets`	Bucket data (members, primary providers)
`StorageAgreements`	Per-provider contracts
`Challenges`	Challenge records
`NextChallengeIndex`	Challenge ID counter
`LastSweptChallengeBlock`	NEW — cursor for the challenge slashing sweep
`PendingChallenges`	Challenges due for slashing sweep
`PendingChallengesByBucket`	Per-bucket challenge index
`ChallengerStats`	Challenger accounting
`MemberBuckets`	Bucket IDs indexed by member

AgreementRequests removed — pending requests no longer live on-chain (off-chain negotiation).
CheckpointConfigs, LastCheckpointWindow, CheckpointRewards, CheckpointPool removed with provider-initiated checkpoints (#306/#311).

2. Off-Chain (Provider Node)

Feature	Status	Notes
Upload/Download	✅	Content-addressed chunks
Commit to MMR	✅	Proper MMR implementation
Read Operations	✅	Byte-range reads
MMR / Chunk Proofs	✅	Real proofs with peak tracking
Provider Signatures	✅	sr25519 via keyfile
Challenge Responder	✅	Auto detection + response
Checkpoint Signing	✅	Client-driven — `GET /checkpoint-signature`
Checkpoint Coordinator	❌	Removed (#306/#311) — provider-initiated coordination / leader election gone
Checkpoint Trigger API	❌	Removed (#306/#311) — `/checkpoint/trigger`, `/checkpoint/duty` gone
Off-Chain Agreement Negotiation	✅	`crates/providers/negotiation` — off-chain handshake before on-chain commitment
Chain Access	✅	`crates/providers/chain` (connection, event decoding) — replaces `chain_state_coordinator.rs`
Replica Sync Coordinator	✅	Full chain integration
S3-Compatible API	✅	PUT/GET/HEAD/DELETE, list, index root
File System API	✅	file CRUD, mkdir, ls, index root
S3 / FS Index Managers	✅	Off-chain metadata with optional JSON persistence
Auth Module (RBAC + signing)	✅	sr25519 request signing, role checks, membership cache with TTL
Persistent Storage	✅	Disk + in-memory backends
Multiaddr On-Chain Sync	✅	Auto-sync bind address on startup
CLI Configuration	✅	Grouped args (rpc/storage/checkpoint/replica/challenge)
Latency-Based Selection	❌	Not implemented

Provider Crates & Node Modules

The provider node was split into crates/providers/* workspace crates — the provider-node binary is now a thin HTTP/wiring layer over them:

crates/providers/auth — request signing, membership, http_auth, verify (replaces the old auth.rs)
crates/providers/chain — chain connection + event decoding (replaces chain_state_coordinator.rs / subxt_client.rs)
crates/providers/challenge — challenge proof building/response
crates/providers/coordinator — provider coordination
crates/providers/negotiation — off-chain agreement handshake (replaces agreement_coordinator.rs)
crates/providers/storage — storage backends, mmr, merkle, nonce

provider-node/src/ on dev: api.rs, challenge_proofs.rs, cli.rs, command.rs, error.rs, fs_api.rs, membership.rs, negotiate.rs, replica_sync.rs, replica_sync_coordinator.rs, s3_api.rs, subxt_client.rs, types.rs, lib.rs, main.rs.

checkpoint_coordinator.rs / chain_state_coordinator.rs removed (provider-initiated checkpoints gone #306/#311; chain access moved to crates/providers/chain).

HTTP Endpoints (~29 routes)

Core: /health, /info, /stats, /node (GET/PUT), /exists, /commit, /read, /commitment, /checkpoint-signature, /mmr_proof, /chunk_proof, /buckets, /delete, /mmr_peaks, /mmr_subtree, /fetch_nodes

Provider-initiated checkpoint routes /checkpoint/sign, /checkpoint/duty, /checkpoint/trigger removed (#306/#311); only the client-driven /checkpoint-signature remains.
Replica: /replica/historical_roots, /replica/sync_status
S3: /s3/:bucket_id/object (PUT/GET/HEAD/DELETE), /s3/:bucket_id/objects, /s3/:bucket_id/index_root
FS: /fs/:bucket_id/file (PUT/GET/DELETE), /fs/:bucket_id/mkdir, /fs/:bucket_id/ls, /fs/:bucket_id/index_root

3. SDKs

TypeScript SDK — primary (#160), `packages/`

Package	Purpose
`@web3-storage/core`	Browser-safe primitives: byte/hex utils, HTTP fetch+retry, base64, provider request signing, CID verification. No chain coupling
`@web3-storage/layer0`	Layer-0 chain binding: PAPI typed wrappers per pallet, signers, in-block tx submission, `watchValue` waits, provider HTTP; `pallet_revive` helpers on `./revive`
`@web3-storage/layer1`	`FileSystemClient` (drives) + `S3Client` (buckets/objects) over layer-0 binding
`@web3-storage/papi`	Single source of truth for runtime descriptors (only tracked metadata snapshot); UIs consume via `workspace:*`
`@web3-storage/sdk`	Umbrella SDK: re-exports core/layer0/layer1 + `Web3Storage` facade

Rust Client SDK — `client/` (still present)

Feature	Status
Upload/Commit/Read, Checkpoint Manager, Checkpoint Persistence, Event Subscription, Provider Discovery (0-100 scoring), Provider Health Tracking, Challenger Client, StorageUserClient, Admin Ops, Data Verification	✅
Automated Spot-Checking / Background Sampling / Multi-Provider Reads	❌

4. Layer 1: S3-Compatible Interface

Feature	Status	Notes
S3 Primitives	✅	`S3BucketInfo`, `ObjectMetadata`, `MetadataEntry`, bounded types
S3 Registry Pallet	✅	5 extrinsics (was 6)
Object Metadata On-Chain	✅	`put_object_metadata`, `delete_object_metadata`, `copy_object_metadata`
S3 Client SDK (Rust + TS)	✅	subxt (Rust) / `S3Client` (TS)
S3 HTTP Endpoints	✅	object CRUD, list, index_root
S3 Index Manager	✅	Off-chain metadata, optional JSON persistence
S3 CI Integration Test	✅	`ci_integration_test.rs`
S3 Precompile	✅ NEW	Solidity-facing precompile

S3 Pallet Extrinsics (5)

create_s3_bucket, delete_s3_bucket, put_object_metadata, delete_object_metadata, copy_object_metadata

create_s3_bucket_with_storage removed — bucket+agreement setup now handled through the off-chain negotiation + establish_storage_agreement path. Read access via runtime API (get_bucket_by_name, get_object, is_bucket_owner, get_layer0_bucket_id).

5. Layer 1: File System Interface

Feature	Status	Notes
FS Primitives	✅	`DriveInfo`, `DirectoryNode`, `FileManifest`, `FileChunk`, `EntryType`
Protobuf + SCALE Dual Encoding	✅	SCALE on-chain, Protobuf off-chain
Drive Registry Pallet	✅	4 extrinsics
Drive Sharing	✅	`share_drive`, `unshare_drive`
FS Client (Rust + TS)	✅	`FileSystemClient`
FS HTTP Endpoints	✅	file CRUD, mkdir, ls, index_root
FS Index Manager	✅	Off-chain file/dir metadata
Content Addressing	✅	CID computation + verification
Drive Registry Precompile	✅ NEW	Solidity-facing precompile

FS Drive Registry Pallet (4 extrinsics)

create_drive, delete_drive, share_drive, unshare_drive (read access via runtime API: get_drive, list_user_drives, is_drive_owner)

Design note: FS pallet stays minimalist — file/dir metadata off-chain (mirrors S3). Only drive registration + Layer-0 bucket mapping are on-chain.

6. Smart Contracts (`pallet_revive` Precompiles) — NEW section

Precompile	Purpose
`storage-provider-precompile`	Bucket/agreement lifecycle exposed to Solidity
`drive-registry-precompile`	Drive lifecycle exposed to Solidity
`s3-registry-precompile`	S3 bucket/object lifecycle exposed to Solidity

pallet_revive (PolkaVM) is wired into both runtimes. Example dApp: examples/contracts/StorageMarketplace.sol.

7. Runtime Configuration

Runtimes (2)

web3-storage-local — Zombienet/local testnet
web3-storage-paseo — Paseo testnet (new)

Configured Pallets (~23)

System, ParachainSystem, Timestamp, ParachainInfo, Balances, TransactionPayment, Sudo, Authorship, CollatorSelection, Session, Aura, AuraExt, XcmpQueue, PolkadotXcm, CumulusXcm, MessageQueue, Utility (new), WeightReclaim (new), Revive (new), StorageProvider, DriveRegistry, S3Registry

Token & Block Parameters

Parameter	Value
Token decimals	12 (`UNIT = 1e12`)
Existential deposit	`MILLIUNIT`
Block hash count	4096
Parachain ID	4000
Relay chain (local)	westend-local

8. User Interfaces

Vite/TS apps under user-interfaces/, consuming @web3-storage/* workspace packages.

App	Package	Notes
Drive UI	`@web3-storage/drive-ui`	Drive browser, upload, folders, access control
Provider Dashboard	`provider-dashboard`	Registration, agreements, checkpoints, challenges, earnings
S3 UI	`@web3-storage/s3-ui`	S3 bucket/object management, challenge flow
Photos	`@web3-storage/photos`	Photos app prototype (#152, #229)
Landing	(static)	Static landing page with `inject-config.mjs` build-time substitution

console-ui is legacy/dead — no package.json/src, only stale build artifacts. Candidate for removal.

Shared: user-interfaces/shared/ + the @web3-storage/* packages (SDK now lives in packages/, not under user-interfaces/sdk).

9. CI/CD & DevOps

GitHub Actions Workflows (11)

Workflow	Purpose
`check.yml`	fmt (nightly), taplo, zepter, check, clippy, tests
`integration-tests.yml`	Build runtime+provider, Zombienet, 2 providers, L0/FS/S3 demos
`check-runtime-migration.yml`	NEW — try-runtime migration check
`security.yml`	NEW — security checks (incl. gitleaks)
`zizmor.yml`	NEW — GitHub Actions security linting
`ui-checks.yml`	NEW — UI lint/build/test
`deploy-ui.yml`	NEW — UI deployment
`release.yml`	NEW — release automation
`cmd.yml` / `cmd-run.yml`	NEW — `/cmd` command bot
`set-image.yml`	Shared CI image selection

Toolchain (`.github/env`)

Rust stable 1.93.0 · Polkadot SDK polkadot-stable2603 · Zombienet v0.4.11 · try-runtime v0.10.1 · Node 22 · solc 0.8.34 · resolc 1.1.0 · gitleaks 8.29.0

Scripts

build-chain-spec.sh, build-paseo-chain-spec.sh (new), check-chain.sh, quick-test.sh, paseo-lower-timeouts-setstorage.mjs (new), cmd/ (new), runtimes-matrix.json (new)

10. Test Coverage

Package	Tests (`#[test]` + `#[tokio::test]`)
`pallet-storage-provider`	~225
`storage-provider-node`	~248
`storage-client` (Rust)	~128
`file-system` (primitives + pallet + client)	~27
`s3` (primitives + pallet + client)	~13
`primitives`	~14

Total: ~650+ Rust tests (up from ~201). Plus TS package/UI tests and Playwright suites in the UIs, and the L0/FS/S3 CI integration flows.

11. Design Phases vs Implementation

Phase	Description	Status
Phase 1	Buckets and Basic Storage	✅ core working
Phase 2	Challenges and Guarantees	✅ now ~95% — cost-sharing sliding scale + batched slashing landed (#125)
Phase 3	Proof-of-DOT	❌ 0% — not started
Phase 4	Third-Party Providers & Replicas	✅ ~90% — full chain integration, RBAC, off-chain negotiation

12. Work Remaining

Resolved since last revision ✅

Challenge cost-sharing sliding scale (was 🔴 Critical) — #125
Burn option — EndAction::Burn
Benchmarks for all extrinsics
Off-chain agreement negotiation — #207

Still Open

Task	Priority	Status
External security audit	🔴 Critical	❌ (in-repo hardening #215 + gitleaks/zizmor tooling added, but no external audit)
Proof-of-DOT identity layer (Phase 3)	🟡 Medium	❌
Automated spot-checking / background sampling	🟡 Medium	❌
Multi-provider reads with failover	🟡 Medium	❌
Latency-based provider selection	🟢 Low	❌
`console-ui` cleanup (dead directory)	🟢 Low	⚠️ housekeeping

13. Conclusion

Production Readiness

⚠️ TESTNET READY — Paseo runtime added; the two March-2026 "critical" blockers (challenge cost scaling, burn) are now implemented. External security audit remains the main gate to production.

Completed:

✅ Core storage (upload/download/commit/MMR/proofs) with ~29 HTTP endpoints
✅ sr25519 + RBAC auth; off-chain agreement negotiation with on-chain commitment
✅ Provider coordination (client-driven checkpoints, challenges w/ sliding-scale cost split + batched slashing, replica sync)
✅ TypeScript SDK (packages/core|layer0|layer1|papi|sdk) + Rust client SDK
✅ Layer 1 File System + S3 interfaces (minimalist pallets + off-chain metadata)
✅ 3 pallet_revive precompiles (storage-provider, drive-registry, s3-registry) + example Solidity dApp
✅ 3 custom pallets: storage-provider (27 ext), drive-registry (4 ext), s3-registry (5 ext)
✅ 2 runtimes (local + paseo), ~23 pallets incl. XCM + Revive
✅ FRAME benchmarking + weights
✅ ~650+ Rust tests + UI/Playwright suites
✅ 5 UIs: drive-ui, provider dashboard, s3-ui, photos, landing
✅ 11 CI workflows incl. try-runtime migration, security linting, release, UI deploy

Required for Production:

❌ External security audit
❌ Proof-of-DOT (Phase 3)
⚠️ Optional robustness: spot-checking, multi-provider read failover, latency-based selection

Last updated: September 2026 (re-verified against origin/dev; prior refresh July 2026 @ 9edd13f).

Scalable Web3 Storage: Gap Analysis

Description