Per process death and restart

Currently the fault type supported by Antidote is only killing an entire node, then restarting it will load the state back from disk and restart all connections (see tests here for examples https://github.com/SyncFree/antidote/blob/master/test/multiple_dcs_node_failure_SUITE.erl). Differently, if a single process within the node dies and restarts it will not necessarily load the state correctly. Fixing this should be easy and should simplify the entire node restart. (Note that transaction coordinator failure during transaction execution is not discussed here since that would require its own recovery process/protocol, here I am only considering the persistent processes running the db).

Here is a list of the processes and what is missing/working:

Stable time calculation (https://github.com/SyncFree/antidote/blob/master/src/meta_data_sender.erl and https://github.com/SyncFree/antidote/blob/master/src/meta_data_manager.erl): This process doesn't need to recover any data since the stable time is calculated from other processes sending it values. The only danger here is that if a process adds a stable time and it is missing an entry for a DC, the time for that DC will not be calculated correctly (see https://github.com/SyncFree/antidote/blob/master/src/stable_time_functions.erl#L42). A simple fix would be for example to set the entry for that DC to 0 until it's safe.

inter_dc_log_sender_vnode : This process propagates transactions to other DCs. The only sate it needs it the id of the last transaction it has sent. On start it waits for the logging_vnode to send this data, instead it could fetch this data itself. This process also sends heartbeats to other DCs based on a timer, it also waits for the logging vnode to start this timer (https://github.com/SyncFree/antidote/blob/master/src/inter_dc_log_sender_vnode.erl#L75). Instead it could start its own timer once it knows the values of the ids previously mentioned.

inter_dc_sub_buf / inter_dc_sub_vnode : This process receives transactions from other DCs, it needs to know how the id of the last transaction received from each DC. It loads this from the logging vnode (https://github.com/SyncFree/antidote/blob/master/src/inter_dc_sub_buf.erl#L56) so no change is needed here.

logging_vnode : The only in-memory state kept here is the id of the last transaction received from each DC and the id of the last transaction committed at this DC. It loads this state on restart, but after restart it only keeps the local transaction id counter up to date. It should also keep the external counter up to date as well, so that when the inter_dc_sub_buf requests this id it gives the correct value.

materializer_vnode : The state of the objects is loaded from the log on process restart, so no change is needed here.

inter_dc_sub : This process connects to other DCs through ZMQ. It does not reconnect on restart, instead it waits for the check_node_restart function to do this (https://github.com/SyncFree/antidote/blob/master/src/inter_dc_manager.erl#L187). It could do this itself.

inter_dc_query : This process handles all other interDC traffic (log recovery, bounded counters, cross DC reads in partial rep). It also waits for the check_node_restart function to do reconnection on restart. It could do this itself.

meta data (https://github.com/SyncFree/antidote/blob/master/src/stable_meta_data_server.erl) : This stores meta-data that needs to be persistent apart from the transactions. For example the name of the local DC and information about other DCs (ips/ports, etc). This information is stored on disk (in dets) and is loaded into memory on process restart so no change is needed here.