Overlay network malfunctions with many nodes

I suffered overlay network malfunctioning for a long time(about 1 years).

Upgrading docker doesn't help me. Even with 18.09.3 docker daemon, the problem exists.

I operate cluster with many nodes (about 1000 machines) and in the cluster docker daemons are installed.
I set up overlay network using zookeeper backend.

When I found overlay network malfunctioning, there are some logs related to "Serf" on docker daemon logs($ journalctl -u docker)

like:

"could not resolve peer \"192.168.13.116\": timed out resolving peer by querying the cluster"

[WARN] serf: Query queue depth (4104) exceeds limit (4096), dropping messages!
[WARN] serf: received old query peerlookup from time

I think high load on Serf makes the problem.
There is no option to configure Serf query queue length as far as I know.
(https://github.com/hashicorp/serf/issues/557)

I usually handle the problem by fixing neighbor table(?) and bridge fdb.

Workaround:

1. Get the right vxlan information. (from zookeeper backend(or etcd or swarm?) and using docker network inspect command)
2. Go inside network namespace on the machine in question.
3. Update neighbor table / bridge fdb

# network_id: ([0-9]+)-{network_hash}
# you can get hash about overlay network using "docker network inspect" command
nsenter --net=/var/run/docker/netns/{network_id?}
# for example:
## remove wrong information / insert right information
ip neigh del 10.1.52.30 lladdr 02:42:0a:01:34:1e dev vxlan0
ip neighbor add 10.1.52.30 lladdr 02:42:0a:01:34:1e dev vxlan0
bridge fdb del 02:42:0a:01:34:1e dev vxlan0 self
bridge fdb add 02:42:0a:01:34:1e dev vxlan0 self dst 10.1.2.3 port 4789
# after doing it, ping between containers works.

No source file or test is named. Start with the Docker daemon logs from journalctl -u docker, the Serf warnings, and the Zookeeper-backed overlay state; compare docker network inspect with the VXLAN neighbor table and bridge FDB. Done should mean identifying and correcting the large-cluster malfunction so container connectivity works without the described manual repair.