moby / moby/libnetwork

Panic in .addServiceInfoToCluster

Open
#2,352 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
2.2k
Forks
875
PR merge metrics
No merged PRs in 30d

Description

DEBU[2019-03-31T21:50:19.926824663+08:00] EnableService 927df965e8be3bd72f9883bcf493cdf7e8d6624b718f6d37721e4e1837d00326 DONE
DEBU[2019-03-31T21:50:19.985169647+08:00] Releasing addresses for endpoint restaurant_adminer_1's interface on network restaurant_default
DEBU[2019-03-31T21:50:19.985207153+08:00] ReleaseAddress(LocalDefault/192.168.112.0/20, 192.168.112.2)
DEBU[2019-03-31T21:50:19.985253319+08:00] Released address PoolID:LocalDefault/192.168.112.0/20, Address:192.168.112.2 Sequence:App: ipam/default/data, ID: LocalDefault/192.168.112.0/20, DBIndex: 0x0, Bits: 4096, Unselected: 4092, Sequence: (0xe0000000, 1)->(0x0, 126)->(0x1, 1)->end Curr:3
DEBU[2019-03-31T21:50:20.093504432+08:00] EnableService f89ea56075dcef61b99bc98c393843d199605c1dd0b06a12f9c35710be264c5e START
DEBU[2019-03-31T21:50:20.093540331+08:00] EnableService f89ea56075dcef61b99bc98c393843d199605c1dd0b06a12f9c35710be264c5e DONE
DEBU[2019-03-31T21:50:20.201824271+08:00] Programming external connectivity on endpoint nginxproxy_nginx-proxy_1 (881a58873e03e58cc500745c7fc22431f2ebe17b65a983579847e94ba859adba)
DEBU[2019-03-31T21:50:20.201929175+08:00] /sbin/iptables, [--wait -t nat -C DOCKER -p tcp -d 0/0 --dport 80 -j DNAT --to-destination 172.17.0.3:80 ! -i docker0]
DEBU[2019-03-31T21:50:20.203698159+08:00] /sbin/iptables, [--wait -t nat -A DOCKER -p tcp -d 0/0 --dport 80 -j DNAT --to-destination 172.17.0.3:80 ! -i docker0]
DEBU[2019-03-31T21:50:20.205038238+08:00] /sbin/iptables, [--wait -t filter -C DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.3 --dport 80 -j ACCEPT]
DEBU[2019-03-31T21:50:20.206311895+08:00] /sbin/iptables, [--wait -t filter -A DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.3 --dport 80 -j ACCEPT]
DEBU[2019-03-31T21:50:20.208492185+08:00] /sbin/iptables, [--wait -t nat -C POSTROUTING -p tcp -s 172.17.0.3 -d 172.17.0.3 --dport 80 -j MASQUERADE]
DEBU[2019-03-31T21:50:20.209881573+08:00] /sbin/iptables, [--wait -t nat -A POSTROUTING -p tcp -s 172.17.0.3 -d 172.17.0.3 --dport 80 -j MASQUERADE]
DEBU[2019-03-31T21:50:20.498900353+08:00] 32c4b5cf56e5344110014261b7277020a48ac1555f9afd009b0a859975115976 (4a541c6).deleteSvcRecords(nodebb-mongo, 172.29.0.2, <nil>, true) updateSvcRecord sid:32c4b5cf56e5344110014261b7277020a48ac1555f9afd009b0a859975115976
DEBU[2019-03-31T21:50:20.498961326+08:00] 32c4b5cf56e5344110014261b7277020a48ac1555f9afd009b0a859975115976 (4a541c6).deleteSvcRecords(mongo, 172.29.0.2, <nil>, false) updateSvcRecord sid:32c4b5cf56e5344110014261b7277020a48ac1555f9afd009b0a859975115976
DEBU[2019-03-31T21:50:20.498988146+08:00] 32c4b5cf56e5344110014261b7277020a48ac1555f9afd009b0a859975115976 (4a541c6).deleteSvcRecords(e81b851480e8, 172.29.0.2, <nil>, false) updateSvcRecord sid:32c4b5cf56e5344110014261b7277020a48ac1555f9afd009b0a859975115976
ERRO[2019-03-31T21:50:20.499966158+08:00] Container not cleaned up from containerd from previous run  container=f89ea56075dcef61b99bc98c393843d199605c1dd0b06a12f9c35710be264c5e error="id already in use"
ERRO[2019-03-31T21:50:20.499995981+08:00] Container not cleaned up from containerd from previous run  container=4acddae4b457b6791b0e7885202b88640aaf7e12e8aeb18886e16bc738899905 error="id already in use"
ERRO[2019-03-31T21:50:20.589820616+08:00] Container not cleaned up from containerd from previous run  container=4279e6ff00a7a80ef55947879f5c38c1f2ce24c5342292887bd494c3c4c240c2 error="id already in use"
ERRO[2019-03-31T21:50:20.589946683+08:00] Container not cleaned up from containerd from previous run  container=3096f288c37f5daf4bd6f5453e1efec8c264af9098a2f61fa98ebb8833774bd0 error="id already in use"
DEBU[2019-03-31T21:50:20.590131348+08:00] bundle dir created                            bundle=/var/run/docker/containerd/3c0567edf35aaa272f5e1f9ba2fcf261ca2ac075e716506b6a98b3f5eb7dc096 module=libcontainerd namespace=moby root=/sharedfolders/docker/overlay2/3eda6e37e72013c76a48a47452cb2098566b36053729b2a22e846ff8ad953a6a/merged
DEBU[2019-03-31T21:50:20.590144339+08:00] bundle dir created                            bundle=/var/run/docker/containerd/ab736ef0f70ec2b109a02a5a0adbe8d35c0167a9c81072c1ae4aa19d667c8eeb module=libcontainerd namespace=moby root=/sharedfolders/docker/overlay2/3d7464d39798022bac95a99fe5b8016a7422b9f2cba4e99fd163c856dac4dc4c/merged
DEBU[2019-03-31T21:50:20.590151253+08:00] bundle dir created                            bundle=/var/run/docker/containerd/120242892a48a70798089dc493ac33a9263c6ab2a71bc5c4a7af509857023f06 module=libcontainerd namespace=moby root=/sharedfolders/docker/overlay2/2238e27dade9d716a3a06a37ad050b6dacbe5338315c0b7bd3d5a5ce2e815f98/merged
DEBU[2019-03-31T21:50:20.594539980+08:00] bundle dir created                            bundle=/var/run/docker/containerd/f89ea56075dcef61b99bc98c393843d199605c1dd0b06a12f9c35710be264c5e module=libcontainerd namespace=moby root=/sharedfolders/docker/overlay2/9659cd6f0df93e57084915f3d6f3a7b9835feef1dd9c62390a517e9a1e58e8cf/merged
DEBU[2019-03-31T21:50:20.617333323+08:00] EnableService 02ae3796d5b5301f5e5a2e9d70ba92eb52cc39a59dcfa47fca2961118a103e2a START
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x56085ef1a345]

goroutine 758 [running]:
github.com/docker/docker/vendor/github.com/docker/libnetwork.(*endpoint).addServiceInfoToCluster(0xc420cbe580, 0xc420bd7d40, 0x0, 0x0)
        /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/agent.go:586 +0xa5
github.com/docker/docker/vendor/github.com/docker/libnetwork.(*sandbox).EnableService(0xc420bd7d40, 0x0, 0x0)
        /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/sandbox.go:702 +0x16f
github.com/docker/docker/daemon.(*Daemon).ActivateContainerServiceBinding(0xc4208fa1e0, 0xc420a9bb40, 0x19, 0x0, 0x0)
        /go/src/github.com/docker/docker/daemon/container_operations.go:1110 +0x89
github.com/docker/docker/daemon.(*Daemon).connectToNetwork(0xc4208fa1e0, 0xc4200f9b00, 0x56085f6fe3d9, 0x6, 0xc420cfa240, 0xc4210fc200, 0x0, 0x0)
        /go/src/github.com/docker/docker/daemon/container_operations.go:788 +0x9db
github.com/docker/docker/daemon.(*Daemon).allocateNetwork(0xc4208fa1e0, 0xc4200f9b00, 0x0, 0x0)
        /go/src/github.com/docker/docker/daemon/container_operations.go:532 +0xa31
github.com/docker/docker/daemon.(*Daemon).initializeNetworking(0xc4208fa1e0, 0xc4200f9b00, 0x0, 0x0)
        /go/src/github.com/docker/docker/daemon/container_operations.go:945 +0xa2
github.com/docker/docker/daemon.(*Daemon).containerStart(0xc4208fa1e0, 0xc4200f9b00, 0x0, 0x0, 0x0, 0x0, 0x1, 0x0, 0x0)
        /go/src/github.com/docker/docker/daemon/start.go:150 +0x2c2
github.com/docker/docker/daemon.(*Daemon).restore.func2(0xc4211dbb30, 0xc4208fa1e0, 0xc420a28a50, 0xc4200f9b00, 0xc420aa7b60)
        /go/src/github.com/docker/docker/daemon/daemon.go:455 +0x2d4
created by github.com/docker/docker/daemon.(*Daemon).restore
        /go/src/github.com/docker/docker/daemon/daemon.go:435 +0x1025

I was fuzzing about configurating Docker Swarm overlay network to electing multiple managers and this happened.

My main manager now ends up now in a deadloop because of this.

I only had a very little hand in Golang, and I don't know what to do.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at vendor/github.com/docker/libnetwork/agent.go:586 and follow the call chain through sandbox.go:702 and daemon/container_operations.go. Reproduce the failure with the reported Swarm overlay network and multiple managers, then verify that container startup no longer panics or leaves the manager in a dead loop.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.