ingress failures: multiple marks for one port

Testing on a 600-node swarm cluster, we see that the ingress network often does not work. e.g.

docker@ip-172-31-3-112:~$ docker service ls | grep :8000
wbns7pnrp777        f1-Stress-Service0            replicated          4/4                 test-image:latest              *:8000->80/tcp,*:8001->80/tcp

docker@ip-172-31-3-112:~$ docker service ps f1-Stress-Service0
ID                  NAME                   IMAGE               NODE                                          DESIRED STATE       CURRENT STATE         ERROR               PORTS
498j5w0twdgh        f1-Stress-Service0.1   test-image:latest   ip-172-31-15-192.us-east-2.compute.internal   Running             Running 2 hours ago                      
zk35ygc10cuk        f1-Stress-Service0.2   test-image:latest   ip-172-31-31-171.us-east-2.compute.internal   Running             Running 2 hours ago                      
6n3yxzh4peo1        f1-Stress-Service0.3   test-image:latest   ip-172-31-18-170.us-east-2.compute.internal   Running             Running 2 hours ago                      
ypgs0xkpx911        f1-Stress-Service0.4   test-image:latest   ip-172-31-8-28.us-east-2.compute.internal     Running             Running 2 hours ago       

docker@ip-172-31-3-112:~$ curl http://127.0.0.1:8000/
curl: (7) Failed to connect to 127.0.0.1 port 8000: No route to host

Looking in the ingress sandbox, I saw multiple iptables rules matching this port (there were 3, but one disappeared after a bit):

bash-4.4# iptables -vL -t mangle --line-numbers | grep 8000
175     58  3239 MARK       tcp  --  any    any     anywhere             anywhere             tcp dpt:8000 MARK set 0x4331
1195    19  1060 MARK       tcp  --  any    any     anywhere             anywhere             tcp dpt:8000 MARK set 0x4c28

0x4331 = 17201
0x4c28 = 19496

bash-4.4# ipvsadm -l -f 17201
Prot LocalAddress:Port Scheduler Flags
 -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
FWM  17201 rr
 -> ip-10-255-51-51.us-east-2.co Masq    1      0          0        
 -> ip-10-255-51-52.us-east-2.co Masq    1      0          0        
 -> ip-10-255-51-53.us-east-2.co Masq    1      0          0        
 -> ip-10-255-51-54.us-east-2.co Masq    1      1          0         

bash-4.4# ipvsadm -l -f 19496
Prot LocalAddress:Port Scheduler Flags
 -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
FWM  19496 rr
 -> ip-10-255-44-244.us-east-2.c Masq    1      0          0        
 -> ip-10-255-44-246.us-east-2.c Masq    1      0          0    

As there were supposed to be 4 replicas, I assumed the 17201 entry was correct and deleted the other one:

bash-4.4# iptables -t mangle -D PREROUTING 1195
bash-4.4# iptables -vL -t mangle --line-numbers|grep 8000
175     58  3239 MARK       tcp  --  any    any     anywhere             anywhere             tcp dpt:8000 MARK set 0x4331
bash-4.4# 

After that, the curl command worked.

(I found I had to be quick, because the rule numbers seem to change very frequently, which is odd as no services were being created or removed at this point)

We can reproduce this fairly easily, although it may take a few attempts. We are using Docker 17.06.2-ee16, plus @fcrisciani's back-port of networkdb and fixes to the bulk syncs. networkdb sees all 600 nodes as active and SwarmKit sees all workers as up.

If left alone, the problem fixes itself after an hour or two. However, working services may later stop working again.

Reproduce the failure on a large Swarm cluster, then inspect the ingress sandbox with the shown iptables mangle and ipvsadm commands. Trace why multiple marks target one port and verify that duplicate rules no longer appear and ingress requests continue working without manual deletion.