Metaprotocol Example: Basic Forum

Background

We will start to use metaprotocol approach, as described here

https://github.com/Joystream/joystream/issues/1990

in various places of our current chain where the runtime is presently doing the validation. Primary candidates are currently

• forum
• proposal discussion
• video comments & reactions
• if we even keep it: council blog.

We assume that this has been implemented in the runtime:

https://github.com/Joystream/joystream/issues/2919

so we have extrinsics akin to

• WorkingGroup::worker_remark(origin, worker_id, message: Vec<u8>)
• WorkingGroup::lead_remark(origin, message: Vec<u8>)
• Membership::member_remark(origin, member_id, message: Vec<u8>)

which have corresponding events

• WorkingGroup::WorkerRemarked(worker_id, message: Vec<u8>)
• WorkingGroup::LeadRemarked(message: Vec<u8>)
• Membership::MemberRemarked(member_id, message: Vec<u8>)

The main idea here is to have message buses that are already authenticated by the chain, thereby allowing any metaprotococol processing logic to avoid signature verification and custom nonce management.

Purpose

The purpose of this example is to make it very tangible for what exactly it would look like for someone to take this approach.

Example

Domain

Let us start with a very basic kind of forum

• Each thread has
  • a unique (across threads) non-negative integer ID assigned automatically upon creation
  • a title text
  • a body text
  • a member as the author
  • a creation date
• Each post, which is
  • a thread id for the thread in which the post lives
  • a unique (across posts in the same thread) non-negative integer ID assigned automatically upon creation
  • a parent thread id, corresponding to the entity to which the post corresponds.
  • a body text
  • a creation date
• Any post or thread body text can be perpetually edited by the author.
• Each thread or post as at most one reply, so there is no threading.
• Any worker in the forum working group can delete any thread, resulting in the deletion of the corresponding entity.

Messages

Based on the domain above, we can suggest with a rather plausible set of message formats which will be broadcasted using the extrinsics mentioned above, but note

• I am not sure if optional variation still is supported in proto3, if not, some other way is needed to signal that a given field may or may not be set, e.g. typically when wanting to edit an old value. Remember that None != null, because the latter means setting the value to nothing, while None would imply not updating whatever the existing value is.
• A number of the properties outlines in the section prior are not payload values in these messages, they are instead derived by the query node mappings which will process these messages in Hydra.

message CreateThread {
  uin32 message_id = 1;
  string title = 2;
  string body = 3;
}

message EditThread {
  uin32 message_id = 1;
  uint64 thread_id = 2;
  optional string new_title = 3;
  optional string new_body = 4;
}

message CreatePost {
  uin32 message_id = 1;
  uint64 thread_id = 2;
  string body = 3;
}

message EditPost {
  uin32 message_id = 1;
  uint64 thread_id = 2;
  uint64 post_id = 3;
  string new_body = 4;
}

message DeleteThread {
  uin32 message_id = 1;
  uint64 thread_id = 2;
  string rationale = 3;
}

The message id fields are populated by a unique fixed, publicly known, value for each message during serialization. When deserializing a buffer, even if the process succeeds for a given message type, the id must still be verified to verify a match.

Query Node Schema

Ideally, a fully featured forum would have no impact on the schemas as we currently have them in Olympia, with the exception of likely simplifications as a result of being able to drop the

https://github.com/Joystream/joystream/blob/olympia/query-node/schemas/forum.graphql

but here I will include a sussinct version corresponding to the simplified model we have here.

type ForumThread @entity {

  "Runtime thread id"
  id: ID!

  "Author of the forum thread"
  author: Membership!

  "Thread title"
  title: String! @fulltext(query: "threadsByTitle")

  "All posts in the thread"
  posts: [Post!] @derivedFrom(field: "thread")

  "The intial post created along with the thread"
  initialPost: Post

  // missing creation date stuff, is related to events

}

type Post @entity {

  "Runtime post id"
  id: ID!

  "Author of the forum post"
  author: Membership!

  "Thread the post was submitted in"
  thread: ForumThread!

  "Content of the post (md-formatted)"
  text: String! @fulltext(query: "postsByText")

  "A post that this post replies to (if any)"
  repliesTo: ForumPost

  // missing creation date stuff, is related to events
}

There are also a bunch of event entities which should also be maintained, but skipping them for now.

Query Node Mappings

The query node mappings for each of the events above will proceed to attempt to decode message payload with one of the supported messages:

• WorkingGroup::WorkerRemarked looks for
  • DeleteThread
• Membership::MemberRemarked looks for
  • CreateThread
  • EditThread
  • CreatePost
  • EditPost

Be aware that it is entirely possible that message cannot be decoded into any supported message, in which case, it should just be ignored. It is also critical that it is ensured that the right working group is being processed. Another important point is that to make certain kinds of actions feasible, like deleting an entire thread in one message (which previously was not possible on-chain), one must exploit the native filtering capabilities of the underlying database in order to make it computationally efficient. In some case, depending on the use case, this may warrant introducing redundant information that makes this sort of processing faster.

A final concern to be aware of is that significant validation will be required in these mappings, compared to normal mappings. In normal mappings, you can trust that the validation was done by the runtime, so all entities being referenced are known to exist and be accessible to the caller, however now this is no longer the case. This means you cannot blindly trust that, for example, the author issuing EditThread is referring to a thread that actually exists, or that this caller actually is the author for.

DevUX

Notice that when a metatransaction fails, because the processor rejects it, there is no event returned from the full node or the query node. There is also no clear success event. The lack of such an event, and corresponding error code or new IDs created, can be a source of complexity for applications. One can try to introduce metaprotocol events as entities, but they are not sent to the caller via any sort of subscription interface.

There is a way to address this: by introducing subscriptions into Hydra that mappings can write to, but this does not exist. Read more about the DevUX pains of the metaprotocol approach, and ideas for what we can do about them longer term.

https://github.com/Joystream/joystream/issues/2018