moby / moby/moby

iptables DOCKER chain contains an infinitely-looping rule

Open
#43,283 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/networking kind/bug version/20.10
Dominant language
Go
Stars
72.1k
Forks
19.2k
Avg merge
1d 17h
Merged PRs (30d)
164

Description

Description

Under some unknown conditions (for now), dockerd creates buggy iptables rules which keeps the default bridge network from providing internet connectivity.

Steps to reproduce the issue:
1.
2.
3.

Describe the results you received:
$ docker run --rm -ti alpine:3.14 ping -c1 -w 2 8.8.8.8 fails

$ iptables -vL -t nat

Chain DOCKER (4294967279 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MASQUERADE  all  --  any    !docker0  192.168.169.0/24     anywhere            
    0     0 MASQUERADE  all  --  any    !br-602a5a9ba6fa  172.17.0.0/16        anywhere            
    0     0 DOCKER     all  --  any    any     anywhere             anywhere             ADDRTYPE match dst-type LOCAL
    0     0 DOCKER     all  --  any    any     anywhere            !127.0.0.0/8          ADDRTYPE match dst-type LOCAL
    0     0 RETURN     all  --  docker0 any     anywhere             anywhere            
[...]

See the 4294967279 references and the self-referencing rules.

Describe the results you expected:
A working iptables configuration (and consequently container external connectivity)

Additional information you deem important (e.g. issue happens only occasionally):
Other iptables are buggy, eg:

Chain DOCKER-ISOLATION-STAGE-2 (4294967283 references)
num   pkts bytes target     prot opt in     out     source               destination         
1        0     0 MASQUERADE  all  --  any    !docker0  192.168.169.0/24     anywhere            
2        0     0 MASQUERADE  all  --  any    !br-602a5a9ba6fa  172.17.0.0/16        anywhere            
3        0     0 RETURN     all  --  docker0 any     anywhere             anywhere            
4        0     0 RETURN     all  --  br-602a5a9ba6fa any     anywhere             anywhere            
5        0     0 DOCKER-USER  all  --  any    any     anywhere             anywhere            
6        0     0 DOCKER-ISOLATION-STAGE-1  all  --  any    any     anywhere             anywhere          
[...]
13       0     0 DOCKER-ISOLATION-STAGE-2  all  --  docker0 !docker0  anywhere             anywhere            
14       0     0 DOCKER-ISOLATION-STAGE-2  all  --  br-602a5a9ba6fa !br-602a5a9ba6fa  anywhere             anywhere

... with, again, same-chain referencing rules.

This issue reflects the same symptoms expressed in #36151 but none of the proposed solutions works.

Output of docker version:

Client: Docker Engine - Community
 Version:           20.10.12
 API version:       1.41
 Go version:        go1.16.12
 Git commit:        e91ed57
 Built:             Mon Dec 13 11:45:48 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.12
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.12
  Git commit:       459d0df
  Built:            Mon Dec 13 11:43:56 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.12
  GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.7.1-docker)
  scan: Docker Scan (Docker Inc., v0.12.0)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 2
 Server Version: 20.10.12
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc version: v1.0.2-0-g52b36a2
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.19.0
 Operating System: Debian GNU/Linux 11 (bullseye)
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 512MiB
 Name: node96154-env-0105106.jls-sto1.elastx.net
 ID: EEEN:TL6Q:LKS5:G6MI:YGZP:JWKO:MXHF:QGVB:K4MG:SVIP:VCGC:27EH
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):

ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: venet0: <BROADCAST,POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default 
    link/void 
    inet 127.0.0.1/32 scope host venet0
       valid_lft forever preferred_lft forever
    inet 212.230.123.123/32 brd 212.230.123.123 scope global venet0:0
       valid_lft forever preferred_lft forever
    inet 10.50.7.128/16 brd 10.50.255.255 scope global venet0:1
       valid_lft forever preferred_lft forever
    inet6 2a07:2180:0:1::2c00/128 scope global 
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:ae:36:7a:4c brd ff:ff:ff:ff:ff:ff
    inet 192.168.169.1/24 brd 192.168.169.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:aeff:fe36:7a4c/64 scope link 
       valid_lft forever preferred_lft forever
18: br-602a5a9ba6fa: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:7b:94:e4:6e brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global br-602a5a9ba6fa
       valid_lft forever preferred_lft forever
    inet6 fe80::42:7bff:fe94:e46e/64 scope link 
       valid_lft forever preferred_lft forever

ip r

default dev venet0 scope link 
10.50.0.0/16 dev venet0 proto kernel scope link src 10.50.7.128 
10.50.1.0/24 via 10.50.7.128 dev venet0 
10.51.0.0/16 via 10.50.7.128 dev venet0 
10.52.0.0/16 via 10.50.7.128 dev venet0 
172.17.0.0/16 dev br-602a5a9ba6fa proto kernel scope link src 172.17.0.1 
192.168.169.0/24 dev docker0 proto kernel scope link src 192.168.169.1

docker inspect bridge

[
    {
        "Name": "bridge",
        "Id": "ad3d59a9186b47f318b5c7f47658e38b64dc0ec5b3d53768daa69a2a04637b51",
        "Created": "2022-02-23T05:08:03.827705859Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "192.168.169.0/24",
                    "Gateway": "192.168.169.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]

It's Debian 11 bullseye (where iptables is bases upon nftables)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the failed docker run ... ping command on Debian 11 with Docker 20.10.12, then inspect iptables -vL -t nat, the DOCKER chain, and DOCKER-ISOLATION-STAGE-2. Done means Docker creates a non-self-referencing iptables configuration and the default bridge provides external connectivity.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, linux
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.