moby / moby/libnetwork

Docker daemon crash when use custom network and no masquerade

Open
#2,537 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
2.2k
Forks
875
PR merge metrics
No merged PRs in 30d

Description

Hi! Got a mysterious crash of docker daemon when using custom network with several services. I can reproduce this by just two steps:

  1. Create custom network, config like this:
docker network create \
        -d bridge \
        --opt com.docker.network.bridge.enable_ip_masquerade=false \
        --subnet 10.6.4.1/24 \
        --gateway 10.6.4.1 \
        tester
  1. Run any command requires dns lookup: docker run --rm --network tester alpine:latest nslookup docker.com

Crushlog:

panic: runtime error: invalid memory address or nil pointer dereference
 [signal SIGSEGV: segmentation violation code=0x1 addr=0x11 pc=0x55eb9baf337b]
 goroutine 464 [running]:
github.com/docker/libnetwork.(*resolver).ServeDNS(0xc000390000, 0x55eb9d65dec0, 0xc000179bc0, 0xc000cdbd40)
         /build/docker.io-yDS88k/docker.io-19.03.6+dfsg1/.gopath/src/github.com/docker/libnetwork/resolver.go:487 +0x79b
 github.com/miekg/dns.(*Server).serveDNS(0xc000fced80, 0xc000e23800, 0x1f, 0x200, 0xc000179bc0)
         /build/docker.io-yDS88k/docker.io-19.03.6+dfsg1/.gopath/src/github.com/miekg/dns/server.go:609 +0x2e2
 github.com/miekg/dns.(*Server).serveUDPPacket(0xc000fced80, 0xc0007ff5c0, 0xc000e23800, 0x1f, 0x200, 0xc00113e4a0, 0xc000bce960)
         /build/docker.io-yDS88k/docker.io-19.03.6+dfsg1/.gopath/src/github.com/miekg/dns/server.go:549 +0xb4
 created by github.com/miekg/dns.(*Server).serveUDP
         /build/docker.io-yDS88k/docker.io-19.03.6+dfsg1/.gopath/src/github.com/miekg/dns/server.go:479 +0x2ae

Some info that can be helpful:

$ cat /etc/docker/daemon.json 
{
        "ipv6": true,
        "fixed-cidr-v6": "fd00::/64",
        "features": {
                "buildkit": true
        },
        "graph": "/sdata/docker/system"
}
$ docker version
Client:
 Version:           19.03.6
 API version:       1.40
 Go version:        go1.13.8
 Git commit:        369ce74
 Built:             Wed, 26 Feb 2020 11:20:11 +1100
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.6
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.8
  Git commit:       369ce74 
  Built:            Wed Feb 26 00:20:11 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          19.03.6
  GitCommit:        7c1e88399ec0b0b077121d9d5ad97e647b11c870
 runc:
  Version:          1.0.0~rc10+dfsg1
  GitCommit:        1.0.0~rc10+dfsg1-1
 docker-init:
  Version:          0.18.0
  GitCommit:
$ docker info
Client:
 Debug Mode: false

Server:
 Containers: 4
  Running: 4
  Paused: 0
  Stopped: 0
 Images: 38
 Server Version: 19.03.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7c1e88399ec0b0b077121d9d5ad97e647b11c870
 runc version: 1.0.0~rc10+dfsg1-1
 init version: 
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.19.0-5-amd64
 Operating System: Debian GNU/Linux bullseye/sid
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.58GiB
 Name: cloud
 Docker Root Dir: /sdata/docker/system
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at libnetwork/resolver.go:487 using the custom bridge network and DNS lookup commands in the report to reproduce the daemon crash. Trace the resolver state for networks with masquerading disabled, then verify the daemon no longer panics and add a regression test if the existing test structure supports it.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
networking
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.