paritytech / paritytech/web3-storage

Replace inline Bucket.members BoundedVec with BucketMembers StorageDoubleMap

Open
#299 2 comments 2 reactions 1 assignee View on GitHub

@RafalMirowski1 is already working on this.

Since Jul 20, 2026.

Dominant language
Rust
Stars
12
Forks
3
Avg merge
2d 2h
Merged PRs (30d)
33

Description

Problem

Bucket.members is a BoundedVec<Member<T>, T::MaxMembers> stored inline in the Bucket struct. With MaxMembers = 100, that's ~3.3 KB (100 × ~33 bytes) decoded on every Buckets::get — including checkpoint extrinsics, which never touch members. Checkpoints are the hot path; this is avoidable PoV/decode cost on a parachain. (We deliberately keep primary_providers inline for efficient checkpoint reads — inline members undermines that same optimization.)

On top of that, every membership check is a linear search over the vec — not the end of the world at a hundred entries, but far from ideal for what is fundamentally a keyed lookup.

On-chain access patterns are almost entirely point lookups:

  • ensure_admin / ensure_writer_or_admin (auth)
  • private-bucket challenge membership gate
  • last-admin invariant (needs an admin count, not the list)
  • full enumeration only off-chain (runtime API → provider membership cache)

Expected cardinality

Cardinality differs sharply by role:

  • Admins/Writers: inherently few — many writers is a coordination problem regardless (append ordering, checkpoint signing). If these were the only roles, shrinking MaxMembers to ~16 and keeping the inline vec would be the simpler fix.
  • Readers: the read ACL for private buckets. Team-drive and paid-access use cases plausibly need hundreds or more. This is what motivates removing the structural bound rather than tightening it.

Proposal

BucketMembers: StorageDoubleMap<Blake2_128Concat, BucketId, Blake2_128Concat, AccountId, Role>
  • Auth checks become one cheap point read; hot checkpoint path drops ~3.3 KB per bucket read/write.
  • Removes the structural member cap — membership becomes effectively unbounded, sized by deposits instead.

Alternative considered: hybrid — keep Admin/Writer inline (small bound), move only Readers to a map. Readers are only checked on-chain at the private-challenge gate (a point lookup anyway), so the full map is simpler; hybrid adds a second code path for little gain.

Considerations

  1. Storage deposit per member — with the bound gone, each set_member must reserve a deposit (refunded on removal) to prevent state spam.
  2. Counters in Bucket — add member_count: u32 and admin_count: u32, maintained on set/remove. Needed for the last-admin invariant and weight accounting, since iteration is no longer an option.
  3. Bucket deletionclear_prefix over an unbounded set can't run in one extrinsic. Use paged removal (limit + cursor over multiple calls) or lazy reaping (tombstoned bucket, anyone can reap entries for the deposit).
  4. MemberBuckets reverse index — currently Account -> BoundedVec<BucketId>; nothing on-chain reads it. Either keep it (per-account bound stays fine) or drop it and leave account→buckets enumeration to indexers. Deletion/cleanup must be handled either way.
  5. Off-chain enumeration — runtime API switches to prefix iteration; unbounded is fine off-chain. Provider membership cache should handle large reader sets (pagination in the runtime API response may be needed).
  6. Migration — storage migration moving existing inline members into the map, plus Bucket struct change.

Affected code

  • pallet/src/lib.rsBucket struct, set_member/remove_member extrinsics
  • pallet/src/impls/members.rsensure_admin, ensure_writer_or_admin, locate_member, set_member_internal, remove_member_internal
  • pallet/src/runtime_api.rs — member enumeration
  • pallet/src/migrations.rs
  • docs/design/scalable-web3-storage-implementation.mdBucket definition (~line 358)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.