docker node promotion of multiple nodes failed on large cluster

Issue: In a large size swarm cluster, e.g. 500 nodes, when I do docker node promote <node1> <node2>. It returns two nodes have been promoted to manager. However, if run docker node ls, it seems like one of the two nodes is still displayed as regular worker. If I ssh into that node supposed to be promoted to a manger and run docker node ls there, it just hangs forever. After while if I perform docker node ls again on the leader node, it shows that failed to promote node is down and reachable.
This problem doesn't happen on smaller clusters.

Docker version:
Client:
Version: 18.01.0-ee-1-tp5
API version: 1.30
Go version: go1.8.3
Git commit: 8188efd
Built: Thu Dec 7 01:32:10 2017
OS/Arch: linux/amd64

Server:
Version: 18.01.0-ee-1-tp5
API version: 1.30 (minimum version 1.12)
Go version: go1.8.3
Git commit: 8188efd
Built: Thu Dec 7 01:31:04 2017
OS/Arch: linux/amd64
Experimental: false

To reproduce:

• testkit create --name test-500 500 0
• ssh docker@<test-node-0>
• docker node promote <test-node-1> <test-node-2>
  -- At this point you will see both nodes have been promoted
• docker node ls
  --You will only see one of the two nodes becomes manager
• ssh into the failed node
• docker node ls on that node will be no responding

Reproduce the failure with testkit create --name test-500 500 0, then follow the node promotion path for docker node promote when promoting two nodes at once. Compare manager membership and docker node ls behavior on the leader and failed node. Done means both nodes become managers and remain responsive in the 500-node cluster.