dns resolver not work inside container after restart

Hi

I have a service started via docker-compose. It has multiple containers and a user defined network of type bridge. A strange issue appeared that the containers could not reach each other by service names after running for hours or days and restarted, neither reach any hostname outside containers. It can only be recovered by restarting docker daemon.

I learned a bit about the name resolving mechanism between the containers on the same user-defined network, and found it relies on a internal dns resolver inside each container and the resolver services on address 127.0.0.11:53. So I looked into containers that worked well and found there are corresponding iptables rules for the address. But when the issue occurred, I could not see these info and the resolver seemed not started.

# iptables -t nat -L -n
...
Chain DOCKER_OUTPUT (1 references)
target     prot opt source               destination
DNAT       tcp  --  0.0.0.0/0            127.0.0.11           tcp dpt:53 to:127.0.0.11:33673
DNAT       udp  --  0.0.0.0/0            127.0.0.11           udp dpt:53 to:127.0.0.11:36854

Chain DOCKER_POSTROUTING (1 references)
target     prot opt source               destination
SNAT       tcp  --  127.0.0.11           0.0.0.0/0            tcp spt:33673 to::53
SNAT       udp  --  127.0.0.11           0.0.0.0/0            udp spt:36854 to::53

# netstat -an |grep 36854
udp        0      0 127.0.0.11:36854        0.0.0.0:*

So far, based on my investigation, I suspect that the resolver is not started under some particular conditions. But I don't know what are the possible reasons that might cause the resolver not started when starting a container on a user-defined container. Any ideas or insights?

$ docker version
Client:
 Version:      17.03.2-ce
 API version:  1.27
 Go version:   go1.7.5
 Git commit:   f5ec1e2
 Built:        Tue Jun 27 02:21:36 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.2-ce
 API version:  1.27 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   f5ec1e2
 Built:        Tue Jun 27 02:21:36 2017
 OS/Arch:      linux/amd64
 Experimental: false

No source files or tests are named. Start by reproducing the failure with docker-compose on a user-defined bridge network, then compare 127.0.0.11 resolver and iptables state before and after container restarts. Done means identifying the condition that prevents resolver setup and verifying a correction without requiring a Docker daemon restart.