Documentation: Messaging documentation is out-of-date since Vitess 6.0 release

The documentation doesn't correctly document the messaging feature as of Vitess 6.0+. there were some major backwards incompatible changes made in the 6.0 release. From the release notes, these things are currently missing from https://vitess.io/docs/reference/features/messaging/

• First Class Citizen

• If the messages are stored in a sharded table, the key must also be the primary vindex of the table is no longer required (DML is required to ACK messages if the primmary vindex is not the primary key)

• Up-to-date schemas and example usage are in https://github.com/vitessio/vitess/blob/master/go/test/endtoend/messaging/main_test.go

• time_scheduled and time_created are no longer required nor automatically populated.

• priority is a new required field. Messages with a lower priority will be processed first.

• Message retention is now measured after time_acked instead of time_scheduled

• New table options vt_min_backoff and vt_max_backoff set bounds on exponential backoff for message retries

• 33% backoff jitter has been added for messages that are getting postponed to prevent thundering herds.

• Users of the messages feature are encouraged to read through the full details in issue #5947 #5948 #6114

Slack conversation: https://vitess.slack.com/archives/C9EDPDJT1/p1599391793002800

Mostafa 4:29 AM
I have a question re using messages in a sharded scenario, how can we atomically update/insert a row and add a message? To achieve this, both must reside in the same shard, given that However, every message needs a uniquely identifiable key. If the messages are stored in a sharded table, the key must also be the primary vindex of the table I am not sure how can we achieve this.

sougou:planetscale: 9:03 AM
In this case, the row's primary vindex and the message's primary vindex must be the same. This will guarantee that the message row is in the same shard as the main row.

derekperkins:nozzle: 10:46 AM
@Mostafa it’s awesome once you get it running
10:47
all of our message tables have a sharding id, which is different per keyspace
10:48
and about 50% of the message ids use sequences, and 50% FK to some other id in the keyspace

Mostafa 11:12 AM
I understand that the row’s primary vindex and the message’s primary vindex must be the same
11:15
for example, updating the user’s row whose primary vindex is the user_id, given that multiple messages can be related to the same user, we cannot use the user_id as the unique key for the message and thus we cannot use it as the primary vindex of the message, this due to the restriction that each message must have a unique key and that key must be the primary vindex of the messages table
11:16
so I am not sure how we can guarantee that the message resides in the same shard as the user

derekperkins:nozzle: 11:20 AM
you might be overthinking the sharding, your table shards just like any normal table
11:20
this is an example message table in our vschema that shards on user_id
"iam__invite_email__msgs": {
"column_vindexes": [
{
"column": "user_id",
"name": "hash"
}
]
}

Mostafa 11:21 AM
would this prevent you from having multiple messages for the same user?

derekperkins:nozzle: 11:22 AM
no, you’re just telling it to shard on user id
11:22
CREATE TABLE iam__invite_email__msgs (
time_scheduled bigint NOT NULL,
id bigint NOT NULL,
time_next bigint DEFAULT NULL,
epoch bigint NOT NULL,
time_created bigint NOT NULL,
time_acked bigint DEFAULT NULL,
user_id bigint NOT NULL,
attributes json DEFAULT NULL,
data varbinary(1200) DEFAULT NULL,
PRIMARY KEY (time_scheduled,id),
UNIQUE KEY id_idx (id),
KEY next_idx (time_next,epoch),
KEY iam__invite_email__msgs__users (user_id),
CONSTRAINT iam__invite_email__msgs_ibfk_1 FOREIGN KEY (user_id) REFERENCES users (user_id),
CONSTRAINT iam__invite_email__msgs_ibfk_2 FOREIGN KEY (id) REFERENCES iam_events (event_id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci ROW_FORMAT=COMPRESSED COMMENT='vitess_message,vt_ack_wait=30,vt_purge_after=86400,vt_batch_size=10,vt_cache_size=10000,vt_poller_interval=30,vt_topic=iam_event_stream_topic'
11:23
this is an old style message table definition, but shows how it might be used in the real world

Mostafa 11:25 AM
in this schema the id is a unique key of the message. The docs state that If the messages are stored in a sharded table, the key must also be the primary vindex of the table , so maybe the docs are outdated, or am I missing something?
11:27
let me rephrase, can we shard the messages table just like any other table? or are there some special restrictions?

sougou:planetscale: 11:34 AM
i believe that comment is outdated
11:34
the primary vindex can be different from the primary key
11:34
The one difference is that you will have to use traditional dmls to ack messages.
11:35
the legacy grpc apis accepted only ids as input, and they required that to match the primary vindex

Mostafa 11:39 AM
awesome! last question, should we use this client https://github.com/vitessio/messages, or is it not maintained anymore?
vitessio/messages
Vitess messages client
Website
https://godoc.org/vitess.io/messages
Stars
4
https://github.com/vitessio/messages|vitessio/messagesvitessio/messages | Mar 1st, 2018 | Added by GitHub

derekperkins:nozzle: 11:47 AM
that hasn’t been updated to use the new messaging table format, speaking of which, the docs are outdated too https://vitess.io/docs/reference/features/messaging/
11:50
there were some major backwards incompatible changes made in the 6.0 release. From the release notes:
Messaging: First Class Citizen + API Improvements
The messaging feature has had several breaking changes and feature additions

• Direct inserts are now supported: INSERT INTO msg_table SELECT col1, col2...
• time_scheduled and time_created are no longer required nor automatically populated.
• priority is a new required field. Messages with a lower priority will be processed first.
• Message retention is now measured after time_acked instead of time_scheduled
• New table options vt_min_backoff and vt_max_backoff set bounds on exponential backoff for message retries
• 33% backoff jitter has been added for messages that are getting postponed to prevent thundering herds.
  Users of the messages feature are encouraged to read through the full details in issue #5947 #5948 #6114

Mostafa 11:51 AM
are there any up-to-date examples?

derekperkins:nozzle: 11:53 AM
https://github.com/vitessio/vitess/blob/e522127d86e80fcb02e0ef5534c16bf8672beeb6/go/test/endtoend/messaging/main_test.go
go/test/endtoend/messaging/main_test.go
/*
Copyright 2020 The Vitess Authors.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
Show more
https://github.com/vitessio/vitess|vitessio/vitessvitessio/vitess | Added by GitHub
11:54
usage hasn’t significantly changed. The main difference is the change in required columns
create table sharded_message(
id bigint,
priority bigint default 0,
time_next bigint default 0,
epoch bigint,
time_acked bigint,
message varchar(128),
primary key(id),
index next_idx(priority, time_next desc),
index ack_idx(time_acked)
) comment 'vitess_message,vt_ack_wait=1,vt_purge_after=3,vt_batch_size=2,vt_cache_size=10,vt_poller_interval=1

Mostafa 11:58 AM
thanks a lot!

Start with the messaging page at docs/reference/features/messaging/ and compare its claims with go/test/endtoend/messaging/main_test.go and the Vitess 6.0 release notes. Update the page with the current schema, sharding and ACK behavior, retention and retry options, and examples, then verify that each checklist item is covered.