raft: Protect state machine from potential duplicate members

If we only consider swarmkit, as is, it is possible to clone a node from an existing state, using the same Raft ID and directly contact the existing members, bypassing the join mechanism.

The cluster can be made unstable this way as a Raft ID should be unique across members. The problem being two occurrences of a node with the same ID interacting with the existing cluster.

This is also a problem for nodes with dynamic addresses: we need to differentiate a node that was cloned with a new address from the one with an address changed at restart time (on VM restart for example).

This is a generic issue to start thinking about additional steps to prevent the state machine to be corrupted by "cloned" members.

Some potential solutions:

• Always call Join on restart and reject any further action if the ID already exists with a different address or fixed criteria (encouraging to always remove the old node with a given ID if we consider it gone). With this solution, an ID can only be used once for a given member.
• Block the messages on ProcessRaftMessage from nodes with a duplicate ID. As long as we suspect that two nodes are reusing the same ID, we block all the messages from that ID and encourage the user to remove this ID.
• Send a ConfChangeUpdate at restart time to make sure the other members always have the latest informations about the restarted raft node. Send a unique regenerated hash from the current configuration and reject the messages from past members using the same ID.

A tentative has been proposed in #1142, but it seems like the approach is lacking and needs more thoughts.

/cc @aaronlehmann

Start by reading the tentative approach in #1142 and trace the Join, ProcessRaftMessage, and ConfChangeUpdate paths described here. Compare how restarted, dynamically addressed, and cloned members are identified. Done requires an agreed mechanism that prevents duplicate Raft IDs from corrupting the state machine, but this issue does not yet define which proposal or tests should establish that behavior.