box: lease RW mode till deadline

Context

I'm implementing a failover agent: a service that monitors tarantool instances and appoints one instance in each replicaset as a master (set read-write mode).

Let's consider a situation, when the agent lost connectivity with the current master.

                 + -------------------- +
                 |       Witness        |
                 + -------------------- +
                 |                      |
                 |       + --- +        |
                 |       |  A  |        |
                 |       + --- +        |
                 |        |   |         |
                 + -------+---+-------- +
                          |   |
                +-xxxxxxx-+   +--------+
                |                      |
+ --------------+----- +        + -----+-------------- +
|          DC1  |      |        |      |   DC2         |   
+ --------------+----- +        + -----+-------------- +
|               |      |        |      |               |   
|               v      |        |      v               |   
|    + ----------- +   |        |   + ------------ +   |  
|    | T1 (master) |<--+--------+-->| T1 (replica) |   |  
|    + ----------- +   |        |   + ------------ +   |  
|            ^         |        |           ^          |   
|            |         |        |           |          |   
+ -----------+-------- +        + ----------+--------- +
             |                              |
             |                              |
     + ------+------------------------------+----- +
     | ................clients.................... |
     + ------------------------------------------- +

On this schema the failover agent A lost connectivity to T1. It should appoint T2 as the new master. However, it should prevent a situation, when two masters exist in a replicaset at the same time.

There are different ways to achieve it. One of possible variants is to lease master mode for N seconds. This way we know a time when T1 will go to the read-only mode. The agent waits for it and appoints the new master (T2) afterwards.

It is implemented now as a fiber on the master instance that sleeps until the given deadline and then calls box.cfg({read_only = true}) if the master appointment was not renewed by the agent.

Problem statement

There is no guarantee that the fiber will be scheduled accurately on the deadline. It can wakes up after some time afterwards: it depends on amount of fibers and workload in them.

It means that some amount of write requests can be processed by the old master after the deadline.

This situation is called 'split brain'.

Proposal

Implement a mechanism that takes the deadline more accurately on the database level.

The idea is to pass the timeout/deadline to the database and let the database check the deadline on write requests on its own.

Thoughts on an implementation

We have a low resolution timer (37d5ac5ac9bdd4db55300b5b31060077edb9f121) -- it sends SIGALRM to itself 100 times per second and calls clock_gettime(CLOCK_MONOTONIC, <...>) inside the signal handler. It should be pretty accurate, because a signal delivery is fast. And information from this timer can be used on a hot path.

Start with the existing master-instance fiber that calls box.cfg({read_only = true}) and review the low-resolution timer at commit 37d5ac5ac9bdd4db55300b5b31060077edb9f121. Trace how SIGALRM and clock_gettime(CLOCK_MONOTONIC, ...) could expose the lease deadline to write requests. Done means the database enforces the deadline accurately enough to prevent writes after lease expiry.