moby / moby/libnetwork

panic in findLBEndpointSandbox

Open
#2,298 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
2.2k
Forks
875
PR merge metrics
No merged PRs in 30d

Description

Found this panic in the journalctl logs while investigating other container restarts. (During restart some running containers were terminated, which was why I was investigating).

Nov 08 08:34:25 host-000009 dockerd[11306]: time="2018-11-08T08:34:25.766619649Z" level=error msg="fatal task error" error="task: non-zero exit (1)" module=node/agent/taskmanager node.id=uh2gr37arkq5hxc6aqge2lf9l service.id=i5zpbufandth734h0krljnf6g task.id=5oz8aynoku5n0poe2s6454p8v
Nov 08 08:34:35 host-000009 dockerd[11306]: panic: runtime error: invalid memory address or nil pointer dereference
Nov 08 08:34:35 host-000009 dockerd[11306]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x557e4cd7ce40]
Nov 08 08:34:35 host-000009 dockerd[11306]: goroutine 623 [running]:
Nov 08 08:34:35 host-000009 dockerd[11306]: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*endpoint).Name(0x0, 0x0, 0x0)
Nov 08 08:34:35 host-000009 dockerd[11306]:         /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/endpoint.go:286 +0x30
Nov 08 08:34:35 host-000009 dockerd[11306]: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*network).findLBEndpointSandbox(0xc420cfa000, 0x0, 0xc42084e000, 0x30, 0x7f16ab6cf6c8)
Nov 08 08:34:35 host-000009 dockerd[11306]:         /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/service_linux.go:72 +0x16b
Nov 08 08:34:35 host-000009 dockerd[11306]: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*network).rmLBBackend(0xc420cfa000, 0xc42219cf80, 0x10, 0x10, 0xc421fe8dc0, 0x101)Nov 08 08:34:35 host-000009 dockerd[11306]:         /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/service_linux.go:179 +0x5a
Nov 08 08:34:35 host-000009 dockerd[11306]: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).rmServiceBinding(0xc420194100, 0xc421f24920, 0x20, 0xc421f24940, 0x19, 0xc420d64ba0, 0x19, 0xc4216f5780, 0x40, 0xc420c40200, ...)
Nov 08 08:34:35 host-000009 dockerd[11306]:         /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/service_common.go:377 +0xc2a
Nov 08 08:34:35 host-000009 dockerd[11306]: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).handleEpTableEvent(0xc420194100, 0x557e4ddec860, 0xc4231a2230)
Nov 08 08:34:35 host-000009 dockerd[11306]:         /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/agent.go:951 +0x1398
Nov 08 08:34:35 host-000009 dockerd[11306]: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).(github.com/docker/docker/vendor/github.com/docker/libnetwork.handleEpTableEvent)-fm(0x557e4ddec860, 0xc4231a2230)
Nov 08 08:34:35 host-000009 dockerd[11306]:         /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/agent.go:320 +0x40
Nov 08 08:34:35 host-000009 dockerd[11306]: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).handleTableEvents(0xc420194100, 0xc421c71fe0, 0xc4202d4830)
Nov 08 08:34:35 host-000009 dockerd[11306]:         /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/agent.go:818 +0x53
Nov 08 08:34:35 host-000009 dockerd[11306]: created by github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).agentInit
Nov 08 08:34:35 host-000009 dockerd[11306]:         /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/agent.go:320 +0x524
Nov 08 08:34:35 host-000009 systemd[1]: docker.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Nov 08 08:34:35 host-000009 systemd[1]: docker.service: Unit entered failed state.
Nov 08 08:34:35 host-000009 systemd[1]: docker.service: Failed with result 'exit-code'.
Nov 08 08:34:35 host-000009 systemd[1]: docker.service: Service hold-off time over, scheduling restart.
Nov 08 08:34:35 host-000009 systemd[1]: Stopped Docker Application Container Engine.
Nov 08 08:34:35 host-000009 systemd[1]: Starting Docker Application Container Engine...

Docker Version 18.06.1:

$ docker version
Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:24:56 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:23:21 2018
  OS/Arch:          linux/amd64
  Experimental:     false

From docker info:

Kernel Version: 4.15.0-1025-azure
Operating System: Ubuntu 16.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.67GiB

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with vendor/github.com/docker/libnetwork/service_linux.go at findLBEndpointSandbox and inspect vendor/github.com/docker/libnetwork/endpoint.go around line 286, using the supplied stack trace as the entry point. Trace how the nil endpoint reaches Name, then verify with relevant libnetwork tests that the reported restart path no longer panics.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go, linux, ubuntu
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.