moby / moby/swarmkit

Ingress routing broke after service update with start first

Open
#2,937 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
3.7k
Forks
676
Avg merge
4d 9h
Merged PRs (30d)
6

Description

Can't access ports after update service with start first (ingress routing). Disabling start first is resolving issue.

Steps to reproduce the issue:

  1. docker service create --name httpd -p 8001:80 httpd:2.4

  2. iptables --list DOCKER-INGRESS -n
    Chain DOCKER-INGRESS (1 references)
    target prot opt source destination
    ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:8001
    ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED tcp spt:8001
    RETURN all -- 0.0.0.0/0 0.0.0.0/0

  3. docker service update --update-order start-first httpd

  4. docker service update --publish-add target=80,published=8002 httpd

  5. iptables --list DOCKER-INGRESS -n (no any port after update)

Chain DOCKER-INGRESS (1 references)
target prot opt source destination
RETURN all -- 0.0.0.0/0 0.0.0.0/0

  1. systemctl restart docker

  2. iptables --list DOCKER-INGRESS -n

Chain DOCKER-INGRESS (1 references)
target prot opt source destination
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:8002
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED tcp spt:8002
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:8001
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED tcp spt:8001
RETURN all -- 0.0.0.0/0 0.0.0.0/0

  1. docker service update --publish-add target=80,published=8003 httpd

  2. iptables --list DOCKER-INGRESS -n

Chain DOCKER-INGRESS (1 references)
target prot opt source destination
RETURN all -- 0.0.0.0/0 0.0.0.0/0

Describe the results you received:

Not working ports, No Ingress chain in iptables.

Describe the results you expected:

port forward should work, Ingress route in iptable for ports.

Output of docker version:

Docker version 19.03.5, build 633a0ea
centos-release-7-7.1908.0.el7.centos.x86_64

Output of docker info:

 docker info
Client:
 Debug Mode: false

Server:
 Containers: 3
  Running: 2
  Paused: 0
  Stopped: 1
 Images: 17
 Server Version: 19.03.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: yxckzh2e1ywppfe8zk56jrc82
  Is Manager: true
  ClusterID: pqxp4ceu8bfd3l29jko9284un
  Managers: 1
  Nodes: 1
  Default Address Pool: 10.0.0.0/8
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: {IP}
  Manager Addresses:
   {IP}:2377
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 3.10.0-1062.4.1.el7.x86_64
 Operating System: CentOS Linux 7 (Core)
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 1.905GiB
 Name: v61-test
 ID: MOJO:D6CD:IWTJ:DBJD:SHIK:5WCE:NDQV:VULI:2PNF:MQZN:27BH:Y7F4
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 55
  Goroutines: 180
  System Time: 2020-01-20T10:45:28.254867753-05:00
  EventsListeners: 2
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  {REGISTRY}:5000
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: API is accessible on http://0.0.0.0:2375 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/engine/security/security/#docker-daemon-attack-surface

Additional environment details (AWS, VirtualBox, physical, etc.):

Test was made on VM (Hyper-V)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the sequence of docker service update commands on Docker 19.03.5 and inspect the DOCKER-INGRESS chain with iptables after each update. Compare the broken start-first behavior with the working restart result on the documented CentOS environment. Done means published ports remain represented in the ingress chain after service updates without restarting Docker.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go, linux
Domain
devops, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.