moby / moby/libnetwork

Port-mapping containers do not start after ungraceful shutdown

Open
#2,448 1 comment 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
2.2k
Forks
875
PR merge metrics
No merged PRs in 30d

Description

This ticket resembles tickets from 2016 and earlier. For various reasons those are closed. My apologies if this is a duplicate or the problem is fixed in a newer version or the bug belongs to a different repository. I must have missed something while browsing the project page.

Abruptly power-cycle a host, and port-mapping containers fail to start. This does not always happen. The docker-proxy will be listening on the relevant port. A workaround is to gracefully restart the host. Depending upon the root cause, the second restart may work sometimes.

The error message is as follows.
"starting container failed: container 3180fc4b8d11cc4970a41c05af3f4062018b7c8c4b5946cf88c56f1abff5c1c6: endpoint join on GW Network failed: driver failed programming external connectivity on endpoint gateway_3180fc4b8d11 (e2843d2a8d7b394f1f0c0cc5213bc4af9cdc4ebb6f837849b3ea069ff82f16a1): Bind for 0.0.0.0:18246 failed: port is already allocated"

The output of netstat is as follows.

/ # netstat -planu
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
...
udp        0      0 :::18246                :::*                                674/docker-proxy

The iptables output on the host contains the following, even though the docker container which listens on the port fails to come up.

root@predixedge:~# iptables -L -t nat --line-numbers
Chain DOCKER (2 references)
num  target     prot opt source               destination         
1    DNAT       tcp  --  anywhere             172.17.0.1           tcp dpt:53 to:127.0.0.2:53
2    DNAT       udp  --  anywhere             172.17.0.1           udp dpt:53 to:127.0.0.2:53
3    RETURN     all  --  anywhere             anywhere            
4    RETURN     all  --  anywhere             anywhere            
5    DNAT       udp  --  anywhere             anywhere             udp dpt:18246 to:172.18.0.2:18246

The output of "docker version" is:

root@predixedge:~# docker version
Client:
 Version:	18.03.0
 API version:	1.37
 Go version:	go1.9.7
 Git commit:	0f1bb35
 Built:	Sat Aug 24 19:38:46 2019
 OS/Arch:	linux/amd64
 Experimental:	false
 Orchestrator:	swarm

Server:
 Engine:
  Version:	18.03.0
  API version:	1.37 (minimum version 1.12)
  Go version:	go1.9.7
  Git commit:	708b068d3095c6a6be939eb2da78c921d2e945e2
  Built:	Sat Aug 24 19:35:44 2019
  OS/Arch:	linux/amd64
  Experimental:	false

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the abrupt host power-cycle with Docker 18.03, then inspect the reported docker-proxy listener, netstat output, and iptables NAT rules. Done means port-mapping containers start normally after an ungraceful shutdown without stale listeners or conflicting port allocations.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go, linux
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.