Promoting 2 nodes in short succession sometimes leave one node unreachable

In the course of writing some end to end tests for docker, I've found that sometimes promoting 2 nodes in short succession sometimes fails. One of the promoted nodes ends up in "Unreachable" state and has to be removed from the cluster and rejoined.

The failure is intermittent. It does not happen every time. The failure may or may not be related to previous promote/demote cycles failing silently.

The code snippet that causes this:

	// now promote 2 workers at the same time
	// first, pick 2 workers
	candidates := workers[0:2]
	// update both of their specs
	for _, candidate := range candidates {
		// get the latest version of the candidate
		c, _, err := cli.NodeInspectWithRaw(testContext, candidate.ID)
		require.NoError(t, err, "error getting latest version of candidate %v", candidate.ID)
		c.Spec.Role = swarm.NodeRoleManager
		err = cli.NodeUpdate(testContext, c.ID, c.Version, c.Spec)
		require.NoError(t, err, "error with node update")
	}

And the result:

ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS
0d1lke3s3zbwnaopuqqhdlull     sanimej-e2e-1       Ready               Active              Unreachable
m3tj6cxmvpkg8th29w6kr4uoc *   sanimej-e2e-0       Ready               Active              Leader
vn4jsthinmzit3ckvvb3ipttx     sanimej-e2e-2       Ready               Active              Reachable

As you can see, one node is Unreachable

/cc @sanimej

Start with the Docker end-to-end test snippet that promotes two workers through NodeInspectWithRaw and NodeUpdate, then examine the promotion and prior promote/demote-cycle behavior. Reproduce repeated promotions in short succession and compare the cluster's manager statuses. Done means both promoted nodes remain Reachable without requiring removal and rejoining.