Docker Swarm, Upgrade from 18.03.1 to 18.06.1 - Ghost networks when removing stacks

Opened initially as https://github.com/docker/for-linux/issues/424 but think it's more approriate here

Expected behavior

Hello, I have a weird behaviour since upgrading from docker 18.3.1 to 18.6.1 and traefik 1.6.4 to 1.6.6 :
• I deploy docker stacks via a docker-compose.yml file which leads to creating a given network per stack and connecting traefik to this stacks (via docker service update --network-add <network> <traefik>) - so far it worked well
• when I deploy a new version of the stack, I remove traefik from the network (docker service update --network-rm <network> <traefik>) and destroy the stack (docker stack rm ...). It removed stack and network. I had to wait a little bit before network is really deleted but it work. Then, I could deploy a create a new version of the stack and add traefik back to the network

I do not update services as there are some dbs in the stack for which I may need to run migrations. The need to be run one only.

Actual behavior

When I redeploy the stack, I do:

if [ $($DOCKER_BIN stack ls --format "{{.Name}}" |grep ^${COMPOSE_PROJECT_NAME}$ |wc -l) -ne "0" ]; then
    $DOCKER_BIN service update --network-rm ${COMPOSE_PROJECT_NAME}_network traefik_traefik
    $DOCKER_BIN stack rm ${COMPOSE_PROJECT_NAME}
    sleep 20
    $DOCKER_BIN network prune --force
fi

$DOCKER_BIN stack deploy --compose-file docker-compose.yml --with-registry-auth ${COMPOSE_PROJECT_NAME}

if [[ ${TRAEFIK} == "docker" ]]; then
    $DOCKER_BIN service update --network-add ${COMPOSE_PROJECT_NAME}_network traefik_traefik
fi

but output is now:

...
Creating service <stack>_backoffice
failed to create service <stack>_backoffice: Error response from daemon: network <stack>_network not found

• But since the upgrade, the network is not well destroyed as traefik seems still in the network (via docker inspect network <network>)

I need to do docker stack rm traefik so that the ghost network vanish as expected and then redeploy traefik from scracth and reconnect it to other instances. Any idea on this ?

I’m trying to downgrade back to 1.6.4 to see if it’s more from docker side or traefik side... => results are the same with 1.6.4

I also noticed that for the stacks I deploy so far I did not set the overlay network as attachable - did not require it for the last 6 months - shoud I add it ? =>result is the same with attachable network.

Steps to reproduce the behavior

• Having a swarm cluster
• Having a docker-compose file with a given network
• Deploy stack
• Connect traefik to it
• Remove traefik to it
• Destroy stack
• Network is still present

sudo docker network inspect <instance_network>
[
    {
        "Name": "<instance_network>",
        "Id": "11xjt1yz38o5vtw40qbzldorx",
        "Created": "2018-08-29T16:58:57.674468644+02:00",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.49.0/24",
                    "Gateway": "10.0.49.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "lb-<instance_network>": {
                "Name": "<instance_network>-endpoint",
                "EndpointID": "eda2b043adb5e211003258e553a0368d5a2f306245c7609793164a8bb3e5ebe7",
                "MacAddress": "02:42:0a:00:31:04",
                "IPv4Address": "10.0.49.4/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4123"
        },
        "Labels": {
            "com.docker.stack.namespace": "<instance>"
        },
        "Peers": [
            {
                "Name": "517efbb11671",
                "IP": "172.16.0.5"
            }
        ]
    }
]

Output of docker version:

Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:24:56 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:23:21 2018
  OS/Arch:          linux/amd64
  Experimental:     false

Output of docker info:

Containers: 129
 Running: 123
 Paused: 0
 Stopped: 6
Images: 129
Server Version: 18.06.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: xqm2e3fld2vem3xali1795ug1
 Is Manager: true
 ClusterID: 7pf90t57w3oog500hniyt9rgr
 Managers: 1
 Nodes: 4
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Autolock Managers: false
 Root Rotation In Progress: false
 Node Address: 172.16.0.5
 Manager Addresses:
  172.16.0.5:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-134-generic
Operating System: Ubuntu 16.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 62.82GiB
Name: swarm1.*******.net
ID: SAAO:VFA7:YFS4:23ZK:TETY:LINA:ZOFO:URPG:5JYE:3SMU:YP4A:DZZV
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

4 VMswith each 8vCPU / 64 Go RAM

Start with the docker-compose.yml stack deployment and the listed docker service update, docker stack rm, network inspect, and network prune commands. Reproduce the sequence on a Swarm cluster and inspect why the overlay network still contains the Traefik endpoint. Done means removing the stack removes its network cleanly and a subsequent stack deploy succeeds.