docker network ls slow when using overlay

Hi,
In my current setup, 7 nodes, setup with a consul KV to enable overlay network, the docker network ls command take around 10 sec to respond.
Actually i've around 20 overlay networks setup on my cluster, and something like 70 containers.
My nodes have different setup
Kernel 3.14 / Docker 1.12.1
Kernel 4.6.4 / Docker 1.11.2

To try to understand why its take so long to get docker network ls to respond, i monitored consul server in debug mode to see what kind of request are made.

Just for network listing, docker seems to send around 1300 http request to my consul kv store, which seems to be quite a lot ... taking account at least the network latency multiplied by 1300 call ...

The http call to consul are always in the same form

2016/09/02 07:54:51 [DEBUG] http: Request /v1/kv/network/docker/network/v1.0/network/50fe87869127a8eec98ee32f2246d5982c6b956a60c8fd6a6e6eb3eb11b336c8/?consistent= (163.683µs) 2016/09/02 07:54:51 [DEBUG] http: Request /v1/kv/network/docker/network/v1.0/endpoint_count/50fe87869127a8eec98ee32f2246d5982c6b956a60c8fd6a6e6eb3eb11b336c8/?consistent= (162.414µs) 2016/09/02 07:54:51 [DEBUG] http: Request /v1/kv/network/docker/network/v1.0/endpoint/50fe87869127a8eec98ee32f2246d5982c6b956a60c8fd6a6e6eb3eb11b336c8/730a1e69bbedd7e48c6b36180c603ded1c55833b261a0251161c2c3c97c3535f/?consistent= (167.802µs) 2016/09/02 07:54:51 [DEBUG] http: Request /v1/kv/network/docker/network/v1.0/network/50fe87869127a8eec98ee32f2246d5982c6b956a60c8fd6a6e6eb3eb11b336c8/?consistent= (162.063µs) 2016/09/02 07:54:51 [DEBUG] http: Request /v1/kv/network/docker/network/v1.0/endpoint_count/50fe87869127a8eec98ee32f2246d5982c6b956a60c8fd6a6e6eb3eb11b336c8/?consistent= (145.088µs) 2016/09/02 07:54:51 [DEBUG] http: Request /v1/kv/network/docker/network/v1.0/endpoint/50fe87869127a8eec98ee32f2246d5982c6b956a60c8fd6a6e6eb3eb11b336c8/730a1e69bbedd7e48c6b36180c603ded1c55833b261a0251161c2c3c97c3535f/?consistent= (158.063µs) 2016/09/02 07:54:51 [DEBUG] http: Request /v1/kv/network/docker/network/v1.0/network/50fe87869127a8eec98ee32f2246d5982c6b956a60c8fd6a6e6eb3eb11b336c8/?consistent= (156.203µs) 2016/09/02 07:54:51 [DEBUG] http: Request /v1/kv/network/docker/network/v1.0/endpoint_count/50fe87869127a8eec98ee32f2246d5982c6b956a60c8fd6a6e6eb3eb11b336c8/?consistent= (168.111µs) 2016/09/02 07:54:51 [DEBUG] http: Request /v1/kv/network/docker/network/v1.0/endpoint/50fe87869127a8eec98ee32f2246d5982c6b956a60c8fd6a6e6eb3eb11b336c8/78f5fa6d6d712cf445592d3bd535c05c38d9390c119aba4ee12c417f02e34d81/?consistent= (208.926µs) 2016/09/02 07:54:51 [DEBUG] http: Request /v1/kv/network/docker/network/v1.0/network/50fe87869127a8eec98ee32f2246d5982c6b956a60c8fd6a6e6eb3eb11b336c8/?consistent= (195.405µs) 2016/09/02 07:54:51 [DEBUG] http: Request /v1/kv/network/docker/network/v1.0/endpoint_count/50fe87869127a8eec98ee32f2246d5982c6b956a60c8fd6a6e6eb3eb11b336c8/?consistent= (148.608µs) 2016/09/02 07:54:51 [DEBUG] http: Request /v1/kv/network/docker/network/v1.0/endpoint/50fe87869127a8eec98ee32f2246d5982c6b956a60c8fd6a6e6eb3eb11b336c8/78f5fa6d6d712cf445592d3bd535c05c38d9390c119aba4ee12c417f02e34d81/?consistent= (169.265µs)

Maybe there is a better approach to request consul to get all the network information for the cluster.

Thx

Start by reproducing the delay with docker network ls in a multi-node overlay setup and inspect the Consul KV request pattern shown in the issue. Trace the network-listing entry point and compare the requests made for each network and endpoint. Done means network listing avoids unnecessary repeated Consul requests and responds faster without losing network information.