Production readiness: Web3 principles compliance and Polkadot SDK standards

Production Readiness: Web3 Principles Compliance & Polkadot SDK Standards

This issue tracks all architectural changes required to align Web3 Storage with Web3 principles ("less trust, more truth") and Polkadot SDK production standards. Each item includes the problem, impact, and proposed solution.

---

Priority Legend

Priority	Meaning
🔴 P0	Blocks production deployment — must fix
🟠 P1	Serious gap — fix before testnet launch
🟡 P2	Important improvement — fix before mainnet
🟢 P3	Good practice — fix when convenient

---

Part A: Web3 Principles Violations

A1. 🔴 Client-Side Encryption (User Sovereignty)

Problem: Data is stored in plaintext on the provider's disk. The provider can read, copy, or leak every byte. Users have zero privacy from the entity storing their data.

Solution: Implement mandatory client-side encryption before upload.

Design:

• Add an encryption module to client/src/ and the Console UI StorageClient
• Use XChaCha20-Poly1305 (AEAD) for symmetric encryption of each chunk before upload
• Key derivation: x25519 key exchange between client keypair and a per-bucket ephemeral key, or a user-supplied passphrase via Argon2id KDF
• Encryption is applied before chunking and hashing — the provider only ever sees ciphertext
• Decryption key is never sent to the provider — stored client-side (optionally backed up encrypted on-chain via a small metadata field)
• The MMR commitment is over encrypted chunks, so challenge/response works unchanged

Files to change:

• client/src/encryption.rs (new) — encrypt_chunk(key, plaintext) → ciphertext, decrypt_chunk(key, ciphertext) → plaintext
• client/src/storage_user.rs — wrap upload/download with encrypt/decrypt
• user-interfaces/console-ui/src/lib/storage.ts — add WebCrypto-based encryption before putObject
• primitives/src/lib.rs — add EncryptionMetadata type (algorithm, nonce, key hint)

Scope: ~500 LOC Rust, ~200 LOC TypeScript

---

A2. 🔴 Download Integrity Verification (Trustlessness)

Problem: When the client downloads data, there is no cryptographic verification that the bytes match the committed hash. A malicious provider can serve garbage.

Solution: Verify content hash on every download and optionally request Merkle proofs.

Design:

• On download: client re-hashes each chunk with blake2-256 and compares against the expected hash from the MMR leaf
• For high-assurance reads: client requests an MMR inclusion proof from the provider (GET /mmr_proof) and verifies the proof against the on-chain MMR root
• Verification is opt-in per read (always hash-check, optionally proof-check) to balance latency vs. security

Files to change:

• client/src/storage_user.rs — add hash verification in download() after receiving bytes
• client/src/verification.rs — add verify_chunk_integrity(data, expected_hash) → bool and verify_mmr_inclusion(chunk_hash, proof, root) → bool
• user-interfaces/console-ui/src/lib/storage.ts — add hash check in downloadS3Object() and downloadFile()
• Provider already serves /mmr_proof and /chunk_proof — no provider changes needed

Scope: ~200 LOC

---

A3. 🔴 Replace Sudo with On-Chain Governance (Decentralization)

Problem: A single sudo key controls the entire chain. No democracy, no council, no referenda. This is antithetical to decentralization.

Solution: Replace pallet-sudo with Polkadot-standard governance pallets.

Design (phased):

Phase 1 — Testnet (minimal governance):

• Add pallet-collective (Technical Committee, 3-5 members)
• Add pallet-membership for committee management
• Route privileged calls (runtime upgrades, parameter changes) through collective majority vote
• Remove pallet-sudo from runtime

Phase 2 — Mainnet (full governance):

• Add pallet-democracy or pallet-conviction-voting + pallet-referenda
• Add pallet-treasury for ecosystem funding
• Add pallet-scheduler for delayed execution
• Allow token holders to vote on parameter changes (MinProviderStake, ChallengePeriod, etc.)
• Implement pallet-preimage for proposal storage

Files to change:

• runtime/src/lib.rs — remove pallet_sudo, add governance pallets, configure origins
• runtime/Cargo.toml — add governance pallet dependencies
• Chain spec — configure initial committee members

Scope: ~300 LOC runtime config. Standard Polkadot SDK pallets, no custom code needed.

---

A4. 🟠 Automated Spot-Checking (Trustlessness)

Problem: Between checkpoints (potentially hours), the provider is unaccountable. No automated verification happens. Challenge mechanism is purely reactive — someone must manually notice a problem.

Solution: Implement background sampling that continuously spot-checks random chunks.

Design:

• Add a SpotChecker background service to the client SDK (similar pattern to CheckpointManager)
• Every N seconds, pick a random leaf index from the MMR range, request the chunk + MMR proof from provider
• Verify: (a) chunk hash matches expected, (b) MMR proof is valid against last checkpoint root
• On failure: automatically submit on-chain challenge (challenge_offchain)
• Configurable: check interval, sample size, auto-challenge toggle

Files to change:

• client/src/spot_checker.rs (new) — SpotChecker struct with start_checking_loop(), check_random_chunk(), auto_challenge()
• client/src/lib.rs — export SpotChecker
• user-interfaces/console-ui/src/lib/storage.ts — optional: periodic background checks via Web Worker

Scope: ~400 LOC

---

A5. 🟠 Metadata Commitment (Data Integrity)

Problem: S3 object listings and FS directory structures are stored only in provider memory/disk. The provider can lie about what files exist, alter names/timestamps, or hide files — none of this is detectable on-chain.

Solution: Commit a metadata Merkle root alongside the data MMR root in checkpoints.

Design:

• Each S3IndexManager and FsIndexManager already computes metadata_merkle_root() — a deterministic hash over sorted entries
• Include this root in the CheckpointProposal: add metadata_root: H256 field
• On-chain: store metadata_root alongside mmr_root in BucketCheckpointed events
• Client can verify: download the full index from provider, compute root locally, compare against on-chain value
• If mismatch: submit a metadata challenge (new challenge variant)

Files to change:

• primitives/src/lib.rs — add metadata_root: H256 to CheckpointProposal
• pallet/src/lib.rs — store metadata_root in checkpoint extrinsics, add challenge_metadata extrinsic
• provider-node/src/checkpoint_coordinator.rs — include state.s3_index.metadata_merkle_root() in proposals
• provider-node/src/s3_index.rs / fs_index.rs — ensure metadata_merkle_root() is deterministic

Scope: ~300 LOC

---

A6. 🟡 Peer-to-Peer Data Layer (Decentralization)

Problem: The entire data plane is HTTP client-server. No peer-to-peer networking, no content routing, no DHT. If a provider goes offline, data is inaccessible even if replicas exist elsewhere.

Solution: Add a libp2p-based data availability layer alongside (not replacing) the HTTP API.

Design (phased):

Phase 1 — Provider-to-Provider Gossip:

• Add libp2p to the provider node for peer discovery between providers serving the same bucket
• Use Kademlia DHT for content-addressed block routing: hash → provider_peer_id
• Replica sync can use libp2p instead of HTTP (more resilient)
• Providers announce their buckets to the DHT on startup

Phase 2 — Client P2P Access:

• Client SDK can resolve content_hash → provider_peer_id via DHT without knowing the provider URL upfront
• Fallback: if primary provider is offline, client automatically finds replicas via DHT
• Multi-provider reads: request same chunk from multiple providers, verify hash, use fastest response

Files to change:

• provider-node/Cargo.toml — add libp2p dependency
• provider-node/src/p2p.rs (new) — libp2p node setup, Kademlia DHT, bitswap-like protocol
• provider-node/src/command.rs — start P2P node alongside HTTP server
• client/src/p2p_resolver.rs (new) — DHT-based provider discovery

Scope: ~1500 LOC. This is a significant feature — consider as a separate milestone.

---

A7. 🟡 Content-Addressed Discovery (Data Availability)

Problem: No way to ask "who has content with hash X?" Client must already know the provider. This prevents failover, redundancy benefits, and data portability.

Solution: On-chain content registry + off-chain DHT (from A6).

Design:

• Lightweight on-chain index: when a provider commits a checkpoint, the MMR root is already stored per-bucket. Add a reverse index: mmr_root → Vec<(bucket_id, provider_id)>
• For chunk-level discovery: use the libp2p DHT from A6 — providers advertise chunk hashes
• Client SDK DiscoveryClient gains: find_providers_for_content(hash) → Vec<ProviderEndpoint>

Depends on: A6 (P2P layer) for full functionality. The on-chain portion can be done independently.

Scope: ~200 LOC on-chain, ~300 LOC client

---

A8. 🟡 Storage Deposits (Economic Security)

Problem: Users can create buckets, S3 buckets, drives, agreements, and objects on-chain with no deposit. This enables state bloat attacks — an attacker can fill chain storage for free.

Solution: Require deposits proportional to on-chain state usage, refunded on cleanup.

Design:

• Define deposit constants in runtime:
  • BucketDeposit: Balance = 10 * UNIT (refunded on delete_bucket)
  • AgreementDeposit: Balance = 1 * UNIT (refunded on claim/end)
  • S3BucketDeposit: Balance = 5 * UNIT (refunded on delete)
  • DriveDeposit: Balance = 5 * UNIT (refunded on delete)
  • ObjectMetadataDeposit: Balance = MILLIUNIT per object (refunded on delete)
• Use T::Currency::reserve() on creation, unreserve() on deletion (pattern already used for provider stake)

Files to change:

• pallet/src/lib.rs — add deposit reserve/unreserve in bucket and agreement creation/deletion
• storage-interfaces/s3/pallet-s3-registry/src/lib.rs — deposit in create_s3_bucket, put_object_metadata
• storage-interfaces/file-system/pallet-registry/src/lib.rs — deposit in create_drive
• runtime/src/lib.rs — configure deposit constants

Scope: ~200 LOC

---

A9. 🟢 Data Portability Protocol (User Sovereignty)

Problem: No standard protocol for migrating data between providers. If a provider exits or is slashed, data must be manually re-uploaded.

Solution: Define a migration protocol and implement provider-to-provider bulk transfer.

Design:

• New endpoint: GET /export/:bucket_id — streams all chunks for a bucket (authenticated, admin-only)
• New endpoint: POST /import/:bucket_id — bulk import chunks from another provider
• Migration flow: (1) create new agreement with new provider, (2) export from old, (3) import to new, (4) verify MMR roots match, (5) end old agreement
• Client SDK: migrate_bucket(bucket_id, new_provider) → MigrationHandle

Scope: ~500 LOC

---

Part B: Polkadot SDK Standard Violations

B1. 🔴 Benchmark Layer 1 Pallet Weights

Problem: The FS drive registry pallet uses #[pallet::weight(10_000)] for all extrinsics — a fabricated value. create_drive performs multiple storage reads, writes, and cross-pallet calls but claims negligible cost. The S3 pallet uses hardcoded constants with zero proof_size. This risks block overweight and chain stalls.

Solution: Add FRAME benchmarking to both Layer 1 pallets.

Design:

• Create storage-interfaces/file-system/pallet-registry/src/benchmarking.rs — benchmark all 4 extrinsics
• Create storage-interfaces/s3/pallet-s3-registry/src/benchmarking.rs — benchmark all 6 extrinsics
• Generate weights with actual DB read/write counts and proof_size
• Wire into runtime with type WeightInfo = pallet_weights::SubstrateWeight<Runtime>

Reference: Follow the same pattern as pallet/src/benchmarking.rs (952 lines, already working).

# Generate weights for FS pallet
cargo run --release --features runtime-benchmarks -- benchmark pallet \
  --pallet pallet-drive-registry --extrinsic '*' --output weights.rs

# Generate weights for S3 pallet
cargo run --release --features runtime-benchmarks -- benchmark pallet \
  --pallet pallet-s3-registry --extrinsic '*' --output weights.rs

Scope: ~400 LOC per pallet (benchmarking + weights)

---

B2. 🔴 Fix on_finalize Unbounded Iteration

Problem: The main pallet's on_finalize iterates over ALL challenges expiring at the current block with no bound. An attacker can create hundreds of challenges expiring at the same block, causing on_finalize to exceed the block weight limit and stall the chain.

Solution: Replace unbounded on_finalize with bounded on_idle processing.

Design:

• Replace on_finalize with on_idle(n, remaining_weight) which respects the remaining weight budget
• Process at most MaxChallengesPerBlock expired challenges per block
• If more remain, leave them in storage for the next block's on_idle
• Add a PendingSettlements storage item for overflow

#[pallet::hooks]
impl<T: Config> Hooks<BlockNumberFor<T>> for Pallet<T> {
    fn on_idle(n: BlockNumberFor<T>, remaining_weight: Weight) -> Weight {
        let max_per_block = T::MaxChallengesPerBlock::get();
        let per_challenge_weight = T::WeightInfo::settle_challenge();
        let mut used = Weight::zero();

        if let Some(mut expired) = Challenges::<T>::get(n) {
            let mut processed = 0u32;
            while let Some(challenge) = expired.pop() {
                if used.saturating_add(per_challenge_weight).any_gt(remaining_weight) {
                    break;
                }
                Self::settle_challenge(challenge);
                used = used.saturating_add(per_challenge_weight);
                processed += 1;
                if processed >= max_per_block { break; }
            }
            if expired.is_empty() {
                Challenges::<T>::remove(n);
            } else {
                Challenges::<T>::insert(n, expired);
            }
        }
        used
    }
}

Files to change:

• pallet/src/lib.rs — replace on_finalize with on_idle
• pallet/src/benchmarking.rs — benchmark settle_challenge for accurate per-item weight
• runtime/src/lib.rs — add MaxChallengesPerBlock constant

Scope: ~100 LOC

---

B3. 🟠 Add Storage Version to All Pallets

Problem: None of the three custom pallets declare a #[pallet::storage_version]. Runtime upgrades that change storage layout will silently corrupt state with no migration path.

Solution: Add storage version annotations and a migration framework.

Design:

// In each pallet's lib.rs:
const STORAGE_VERSION: StorageVersion = StorageVersion::new(1);

#[pallet::pallet]
#[pallet::storage_version(STORAGE_VERSION)]
pub struct Pallet<T>(_);

When storage schema changes in future:

1. Increment version constant
2. Write migration in mod migrations { pub struct V1ToV2; impl OnRuntimeUpgrade for V1ToV2 { ... } }
3. Add migration to runtime's Executive type

Files to change:

• pallet/src/lib.rs — add STORAGE_VERSION and annotation
• storage-interfaces/file-system/pallet-registry/src/lib.rs — same
• storage-interfaces/s3/pallet-s3-registry/src/lib.rs — same

Scope: ~30 LOC

---

B4. 🟠 Fix Non-Sequential Call Indices

Problem: Main pallet has out-of-order indices (index 4 after 5, index 27 after 28) and gaps. FS pallet is missing index 1. This is confusing, error-prone, and violates Polkadot conventions.

Solution: Reorder all call indices to be sequential.

Important: This is a breaking change for any existing chain state. Must be done in a runtime upgrade with careful coordination. For a chain that hasn't launched yet, this is safe to do now.

Files to change:

• pallet/src/lib.rs — reorder all 35 #[pallet::call_index(N)] to 0..34
• storage-interfaces/file-system/pallet-registry/src/lib.rs — reorder to 0..3
• Update integration tests and PAPI descriptors

Scope: ~50 LOC (mechanical change)

---

B5. 🟠 Decouple Layer 1 Pallets via Traits

Problem: S3 and FS pallets call pallet_storage_provider::Pallet::<T>::create_bucket_internal() directly — tight coupling that prevents independent upgrades and testing.

Solution: Define a StorageProviderInterface trait.

Design:

// In primitives/src/lib.rs or a new crate:
pub trait StorageProviderInterface<AccountId, Balance, BlockNumber> {
    fn create_bucket(
        owner: &AccountId,
        min_providers: u32,
    ) -> Result<BucketId, DispatchError>;

    fn request_agreement(
        requester: &AccountId,
        bucket_id: BucketId,
        provider: &AccountId,
        max_capacity: u64,
        duration: BlockNumber,
        payment: Balance,
    ) -> Result<(), DispatchError>;

    fn delete_bucket(
        owner: &AccountId,
        bucket_id: BucketId,
    ) -> Result<(), DispatchError>;
}

• Main pallet implements this trait
• S3 and FS pallets depend on the trait, not the concrete pallet
• Config: type StorageProvider: StorageProviderInterface<...>;

Files to change:

• primitives/src/lib.rs — define StorageProviderInterface trait
• pallet/src/lib.rs — implement trait
• storage-interfaces/s3/pallet-s3-registry/src/lib.rs — use trait instead of direct calls
• storage-interfaces/file-system/pallet-registry/src/lib.rs — use trait instead of direct calls

Scope: ~200 LOC

---

B6. 🟠 Add Custom Runtime APIs

Problem: Complex queries (provider discovery, checkpoint status, bucket membership) require raw storage queries via subxt dynamic dispatch — fragile and untyped. No custom runtime APIs exist.

Solution: Define typed runtime APIs for common queries.

Design:

// In a new crate: runtime-api/src/lib.rs
sp_api::decl_runtime_apis! {
    pub trait StorageProviderApi<AccountId, Balance> where
        AccountId: codec::Codec,
        Balance: codec::Codec,
    {
        fn get_provider_info(account: AccountId) -> Option<ProviderInfoResponse>;
        fn list_bucket_providers(bucket_id: BucketId) -> Vec<ProviderEndpoint>;
        fn get_checkpoint_status(bucket_id: BucketId) -> CheckpointStatusResponse;
        fn list_user_buckets(account: AccountId) -> Vec<BucketSummary>;
        fn get_agreement_details(bucket_id: BucketId, provider: AccountId) -> Option<AgreementDetails>;
    }
}

Files to change:

• New crate: runtime-api/ with trait definitions
• runtime/src/lib.rs — implement the APIs in impl_runtime_apis!
• client/src/substrate.rs — use typed APIs instead of raw storage queries

Scope: ~400 LOC

---

B7. 🟡 Fix std::sync::Mutex in Async Code

Problem: ProviderState.checkpoint_cmd_tx uses std::sync::Mutex to wrap an async channel sender. Holding a std::sync::Mutex across .await points risks deadlocks in tokio runtime.

Solution: Replace with tokio::sync::Mutex or a lock-free alternative.

Design:

• Option A: Use tokio::sync::Mutex<Option<mpsc::Sender<CoordinatorCommand>>>
• Option B (preferred): Use arc_swap::ArcSwapOption<mpsc::Sender<CoordinatorCommand>> for lock-free reads
• Option C: Use std::sync::OnceLock since the sender is set once at startup and never changed

Option C is simplest and correct since set_checkpoint_handle is only called once:

pub struct ProviderState {
    // ...
    pub checkpoint_cmd_tx: std::sync::OnceLock<mpsc::Sender<CoordinatorCommand>>,
}

impl ProviderState {
    pub fn set_checkpoint_handle(&self, handle: &CheckpointCoordinatorHandle) {
        let _ = self.checkpoint_cmd_tx.set(handle.command_sender());
    }
}

Scope: ~20 LOC

---

B8. 🟡 Remove #[allow(deprecated)] Suppressions

Problem: The FS pallet uses #[allow(deprecated)] to suppress warnings about deprecated FRAME patterns instead of fixing them.

Solution: Migrate to current FRAME patterns.

• Update RuntimeEvent configuration to current frame_system::Config patterns
• Replace deprecated weight types with current Weight API
• Update test mock to use current patterns

Scope: ~50 LOC

---

B9. 🟢 Add try-runtime Migration Tests

Problem: The try-runtime feature flag exists but no migration tests are implemented. This means runtime upgrades cannot be safely tested against live state.

Solution: Add try-runtime tests for all pallets.

Design:

#[cfg(feature = "try-runtime")]
impl<T: Config> Pallet<T> {
    fn pre_upgrade() -> Result<Vec<u8>, sp_runtime::TryRuntimeError> {
        let bucket_count = Buckets::<T>::iter().count();
        Ok((bucket_count as u32).encode())
    }

    fn post_upgrade(state: Vec<u8>) -> Result<(), sp_runtime::TryRuntimeError> {
        let old_count = u32::decode(&mut &state[..]).unwrap();
        let new_count = Buckets::<T>::iter().count() as u32;
        ensure!(old_count == new_count, "Bucket count changed during migration");
        Ok(())
    }
}

Scope: ~100 LOC per pallet

---

Implementation Roadmap

Phase 1 — Critical Fixes (Blocks Testnet)

Issue	Effort	Depends On
B1. Benchmark Layer 1 weights	2 days	—
B2. Fix on_finalize unbounded iteration	1 day	—
A2. Download integrity verification	1 day	—
A8. Storage deposits	1 day	—

Phase 2 — Pre-Testnet (Serious Gaps)

Issue	Effort	Depends On
A1. Client-side encryption	3 days	—
A3. Replace sudo with governance	2 days	—
A4. Automated spot-checking	2 days	A2
A5. Metadata commitment	2 days	—
B3. Storage versions	0.5 day	—
B4. Fix call indices	0.5 day	—
B5. Decouple pallets via traits	1 day	—
B6. Custom runtime APIs	2 days	—

Phase 3 — Pre-Mainnet (Important)

Issue	Effort	Depends On
A6. P2P data layer (libp2p)	2 weeks	—
A7. Content-addressed discovery	1 week	A6
A9. Data portability protocol	3 days	—
B7. Fix async mutex	0.5 day	—
B8. Remove deprecated suppressions	0.5 day	—
B9. try-runtime tests	1 day	B3

Total Estimated Effort

• Phase 1: ~5 days
• Phase 2: ~13 days
• Phase 3: ~4 weeks
• Total: ~6-7 weeks of focused work

---

Created: March 2026

Treat this as an umbrella issue rather than a single starter task. Begin by reading the relevant entry points named in the selected section, such as runtime/src/lib.rs, client/src/storage_user.rs, or the Layer 1 pallet benchmarking files. Done requires narrowing one subsection to a separately testable change; the issue does not define one completion criterion for the whole scope.