docker 1.9.1 overlay bind port failure : 7946

docker info

[root@hps104 ~]# docker info
Containers: 22
Images: 89
Server Version: 1.9.1
Storage Driver: devicemapper
Pool Name: docker-253:17-1575356-pool
Pool Blocksize: 65.54 kB
Base Device Size: 107.4 GB
Backing Filesystem: xfs
Data file: /dev/loop2
Metadata file: /dev/loop3
Data Space Used: 2.535 GB
Data Space Total: 107.4 GB
Data Space Available: 104.8 GB
Metadata Space Used: 6.455 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.141 GB
Udev Sync Supported: true
Deferred Removal Enabled: true
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /data/hps/hps_install/data/docker/devicemapper/devicemapper/data
Metadata loop file: /data/hps/hps_install/data/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.107-RHEL7 (2015-10-14)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.3.3-1.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 4
Total Memory: 15.67 GiB
Name: hps104
ID: N7GK:LHWB:324M:5EVU:3KOR:BAMH:FNFL:YCSP:EK3E:7BOW:ZON3:LTJF
WARNING: bridge-nf-call-ip6tables is disabled
Cluster store: zk://10.1.245.102:2181,10.1.245.104:2181,10.1.245.105:2181
Cluster advertise: 10.1.245.104:2375

the docker daemon config file:

[root@hps104 ~]# cat /usr/lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target docker.socket
Requires=docker.socket

[Service]
Type=notify
ExecStart=/usr/bin/docker daemon -g /data/hps/hps_install/data/docker --insecure-registry 0.0.0.0/0 -H unix:///var/run/docker.sock -H tcp://0.0.0.0:2375 --cluster-store=zk://10.1.245.102:2181,10.1.245.104:2181,10.1.245.105:2181 --cluster-advertise=eth0:2375 --userland-proxy=false --storage-driver=devicemapper --storage-opt dm.fs=xfs --storage-opt=dm.use_deferred_removal=true --exec-opt native.cgroupdriver=cgroupfs
MountFlags=slave
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
Restart=always
RestartSec=20

[Install]
WantedBy=multi-user.target

bind port failure : 7946

The phenomenon appears random

the log file show:

normal :
Apr 19 15:48:20 hps104 docker[5677]: time="2017-04-19T15:48:20.399445340+08:00" level=debug msg="Registering ipam provider: default" Apr 19 15:48:20 hps104 docker[5677]: time="2017-04-19T15:48:20.468140733+08:00" level=debug msg="Watch triggered with 3 nodes" discovery=zk Apr 19 15:48:20 hps104 docker[5677]: time="2017-04-19T15:48:20.469378877+08:00" level=info msg="2017/04/19 15:48:20 [INFO] serf: EventMemberJoin: hps104 10.1.245.104\n" Apr 19 15:48:20 hps104 docker[5677]: time="2017-04-19T15:48:20.470105660+08:00" level=debug msg="2017/04/19 15:48:20 [DEBUG] memberlist: Initiating push/pull sync with: 10.1.245.102:7946\n" Apr 19 15:48:20 hps104 docker[5677]: time="2017-04-19T15:48:20.475508678+08:00" level=info msg="2017/04/19 15:48:20 [INFO] serf: EventMemberJoin: hps105 10.1.245.105\n" Apr 19 15:48:20 hps104 docker[5677]: time="2017-04-19T15:48:20.475568560+08:00" level=info msg="2017/04/19 15:48:20 [INFO] serf: EventMemberJoin: hps102 10.1.245.102\n" Apr 19 15:48:20 hps104 docker[5677]: time="2017-04-19T15:48:20.476727350+08:00" level=debug msg="Received user event name:jl 10.1.245.104 fc7019211f62e111b4dea907babf49cccaefa1e5088d3a2449a418bd4dab7e4e 075b706b81a7da634b8aaa4ca1e1f1c028aec27ca42ed0dbff9f4eddc9434f93, payload:join 10.0.0.2 255.255.255.0 02:42:0a:00:00:02\n"

Abnormal situation:

Apr 19 15:44:56 hps104 docker[4165]: time="2017-04-19T15:44:56.785591154+08:00" level=debug msg="Option OptionKVProviderURL: 10.1.245.102:2181,10.1.245.104:2181,10.1.245.105:2181" Apr 19 15:44:56 hps104 docker[4165]: time="2017-04-19T15:44:56.807148408+08:00" level=debug msg="Watch triggered with 2 nodes" discovery=zk Apr 19 15:44:56 hps104 docker[4165]: time="2017-04-19T15:44:56.980504187+08:00" level=info msg="Firewalld running: false" Apr 19 15:44:56 hps104 docker[4165]: time="2017-04-19T15:44:56.985494525+08:00" level=debug msg="/usr/sbin/iptables, [--wait -t nat -D PREROUTING -m addrtype --dst-type LOCAL -j DOCKER]" Apr 19 15:44:56 hps104 docker[4165]: time="2017-04-19T15:44:56.988088256+08:00" level=debug msg="/usr/sbin/iptables, [--wait -t nat -D OUTPUT -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j DOCKER]"

When the abnormal situation appear ,it looks like docker don't load overlay code.

Start with the supplied docker info, systemd unit, and normal versus abnormal daemon logs, focusing on cluster discovery and the 7946 bind. Reproduce startup with the shown Docker 1.9.1 configuration and compare whether the overlay membership process initializes; done means the overlay network consistently loads and port 7946 binds across restarts.