Second manager not restarting

TestApiSwarmManagerRestore starts 1 manager, restarts it, then joins a second manager and restarts it. In some cases (tested on slow arm machine) the second manager does not come up.

In the logs of the first manager I see

time="2016-08-10T22:59:59.484249264Z" level=debug msg="feedb2d64d21678 failed to send message to 666759e4948b39aa because it is unreachable [next = 36, match = 0, state = ProgressStateProbe, waiting = false, pendingSnapshot = 0]" 
time="2016-08-10T23:00:05.580604416Z" level=debug msg="heartbeat expiration" 

https://gist.github.com/tonistiigi/d6c7f505a9f7815c4fdfe22f69420526#file-manager1-log-L507-L524

In second manager(the one that is being restarted)

time="2016-08-10T23:00:02.103352688Z" level=error msg="could not find cluster member to query for leader address" 
time="2016-08-10T23:00:02.104053263Z" level=warning msg="sending message to an unrecognized member ID feedb2d64d21678" 
time="2016-08-10T23:00:02.137347785Z" level=error msg="error reestabilishing connection to leader" error="raft: no elected cluster leader"
time="2016-08-10T23:00:02.464558885Z" level=error msg="could not find cluster member to query for leader address" 
time="2016-08-10T23:00:05.254066131Z" level=error msg="agent: session failed" error="session initiation timed out" module="node/agent"

Also time="2016-08-10T22:59:55.068033302Z" level=info msg="newRaft 666759e4948b39aa [peers: [], term: 3, commit: 0, applied: 0, lastindex: 0, lastterm: 0]" indicates that there is something wrong with raft state as it doesn't include the service previously created.

https://gist.github.com/tonistiigi/d6c7f505a9f7815c4fdfe22f69420526#file-docker-log-L259-L319

Tested with: https://github.com/docker/docker/commit/2a540c18b69bebd7153ba87ffc91aade95ba2769
Swarmkit version: https://github.com/docker/swarmkit/commit/cb6d81316727941665594f153434e5ce2e425c9b

Let me know if you want more debug logs added as it is probably very hard to reproduce locally.

cc @aaronlehmann

Start with TestApiSwarmManagerRestore and reproduce the restart sequence described, using the referenced manager and Docker logs to compare the first and second managers. Inspect the Raft state around newRaft and leader reestablishment; done means the second manager reliably starts with the previously created service and an elected leader.