failover: describe failover options in detail in the platform document

Product: Tarantool EE
Since: --
Audience/target: who will read it and for what task?
Root document: https://www.tarantool.io/en/doc/latest/platform/replication/supervised_failover/
SME: @georgiy-belyanin

Please, note this issue assumes tarantool/tarantool-ee#1697 is implemented in tarantool/tarantool-ee#1699

Details

Some of the options described in the reference sections (see #5269 #4647 #4993) are actually not trivial. It would be nice to provide some more examples apart from the reference for users reading the generic document about the failover to understand the failover behavior better a little bit.

It is suggested to introduce a new section Configuring the failover describe there a few configuration options and supplement it with links to the configuration reference.

TODO: add information about config.storage/etcd configuration, disabling the stateboard, please, ask @georgiy-belyanin

Learners

Tarantool Configuration allows you to mark some instances as learners. The learners can't be appointed by the failover.

This mechanism allows you to guarantee the instance won't be chosen if the other master fails. This is particularly useful for maintence.

Here should go an example from the corresponding issue

Priorities

In case of master failure the failover coordinator appoints the instance that has reported the largest vclock number retrieved from the latest master. In other words, the failover checks instance statuses and tries to estimate which of the instances have received the most transactions from the existing master thus having the least lag trying to avoid transaction conflicts. If there are multiple instances reporting the same vclock number the coordinator chooses the first

Apart from vclock-based heuristics, the failover offers a mechanism to prioritize instances as masters candidates that allows one to make some instances more preferable than others. The configuration maps priorities to instances and the failover prefers non-anonymous alive instances having the largest priority. If not specified the priority is considered to be 0 implicitly. The priority can be negative. Higher explicit priorities override vclock heuristics. If the priorities are equal the coordinator chooses the instance with larger vclocks. Refer to the following examples for the behavior to become clearer.

Note: The priority mechanism doesn't mean that the failover would disappoint masters that have less priority in favor to appoint the one with greater priority. Essentially, it is only a recommendation for the failover coordinator when it is choosing a new instance in case of failure of the previous master. Please, execute switch manually if you want to switch onto a more preferable master.

Priority example 1:

In that case the failover appoints non-anonymous alive instance which has reported the largest vclock number from the last master, the instance that is assumed to have received the most transactions.

failover:
  replicasets:
    replicaset-001:
      priority: {}

Priority example 2:

In that case the failover appoints instance-001. If it is not alive it appoints instance-002. Then the failover chooses between instance-003 and instance-004 preferring the bigger vclock value reported from the last master.

failover:
  replicasets:
    replicaset-001:
      priority:
        instance-001: 5
        instance-002: 4
        instance-003: 3
        instance-004: 3

Priority example 3:

Assume there are three instances instance-001, instance-002, instance-003. In that case the failover chooses from instance-001 and instance-002 w.r.t. reported vclocks. If both these instances are unavailable the failover chooses instance-003. That is, if the priority is not specified it is assumed to be set to 0. Negative priority means that the instance is less desirable than the default ones.

failover:
  replicasets:
    replicaset-001:
      priority:
        instance-003: -1

Synchronous replication

Note: synchronous replication for 3-DC scenarios was added in Tarantool 3.5. For 2-DC and for other scenarios was added in Tarantool 3.6.

If you need synchronous replication along with the failover coordinator it should be enabled explicitly in the configuration.

failover:
  replicasets:
    replicaset-001:
      priority:
        synchro_mode: true

In that case, it is desirable to mark all replicaset-001 spaces as synchronous.

TODO: Describe synchronous in detail, please, ask @georgiy-belyanin

Start with the root document at platform/replication/supervised_failover/ and review the related issues #5269, #4647, and #4993. Consult @georgiy-belyanin about config.storage/etcd, disabling the stateboard, and synchronous replication, then add the proposed configuration section with examples and links to the reference material. Done means the document explains learners, priorities, and synchronous replication clearly, including the listed TODO areas.