Antidote fault tolerance and recovery

Currently Antidote's primary fault tolerance is through updates being replicated at multiple DCs and that lost messages across DCs are detected and resent.

The design does not ensure safety and progress and recovery when:
A process crashes within a DC, whether that be a vnode process, a maintenance process, or a transaction coordinator.
Messages within the DC are lost.
Any type of ring handoff that happens outside of system initialisation when no transactions are happening.
(Please add more as you notice them)

We should come up with an overall design and some guidelines for ensuring fault tolerance that fits well with the supervisor model.

Here is a comment from Marc about the transaction prepare and abort messages that we should use when designing the model:

"the fate of a transaction logically depends ONLY on unanimous "prepare". Once all transaction participants have recorded the prepare message in their log, the transaction IS committed, and all entities in the system must reach this same conclusion no matter what happens. If there is a state where not all participants have recorded prepare, and one knows for sure that this will not happen in the future (e.g. a participant is known to have crashed) then everyone must conclude "abort". The abort message is an optimisation, and the termination should take place even if participants crash or messages get lost."

This thread can be used as point to keep track of where we encounter fault tolerance problems and the discussion fixing them.