moby / moby/moby

SIGSEGV: segmentation violation code=0x2 addr=0xc001302ce9 pc=0x2986024] on dind (`grpc.(*ClientConn).resolveNow`)

Open
#49,285 8 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

kind/bug status/0-triage status/more-info-needed version/27.5
Dominant language
Go
Stars
72.1k
Forks
19.2k
Avg merge
1d 17h
Merged PRs (30d)
164

Description

Description

Upon starting dockerd the process terminates with SIGSEGV: segmentation violation code=0x2 addr=0xc001302ce9 pc=0x2986024]

Full log 
cat: can't open '/proc/net/arp_tables_names': No such file or directory
iptables v1.8.10 (nf_tables)
time="2025-01-16T10:25:23.359110652Z" level=info msg="Starting up"
time="2025-01-16T10:25:23.359609123Z" level=info msg="containerd not running, starting managed containerd"
time="2025-01-16T10:25:23.360487902Z" level=info msg="started new containerd process" address=/var/run/docker/containerd/containerd.sock module=libcontainerd pid=161
unexpected fault address 0xc001302ce9
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x2 addr=0xc001302ce9 pc=0x2986024]
goroutine 57 gp=0xc0005a3500 m=11 mp=0xc0005d2808 [running]:
runtime.throw({0x9000fb?, 0x8000?})
	/usr/local/go/src/runtime/panic.go:1023 +0x5c fp=0xc000575dd8 sp=0xc000575da8 pc=0x213b5dc
runtime.sigpanic()
	/usr/local/go/src/runtime/signal_unix.go:895 +0x285 fp=0xc000575e38 sp=0xc000575dd8 pc=0x2154285
google.golang.org/grpc.(*ClientConn).resolveNow(0xc0005a7008, {})
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:1081 +0x44 fp=0xc000575e60 sp=0xc000575e38 pc=0x2986024
google.golang.org/grpc.(*addrConn).resetTransportAndUnlock(0xc000322c08)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:1254 +0x258 fp=0xc000575f78 sp=0xc000575e60 pc=0x2986a38
google.golang.org/grpc.(*addrConn).connect(0xc000322c08)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:912 +0x145 fp=0xc000575fc8 sp=0xc000575f78 pc=0x2985005
google.golang.org/grpc.(*acBalancerWrapper).Connect.gowrap1()
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/balancer_wrapper.go:300 +0x25 fp=0xc000575fe0 sp=0xc000575fc8 pc=0x297f9a5
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000575fe8 sp=0xc000575fe0 pc=0x2176d41
created by google.golang.org/grpc.(*acBalancerWrapper).Connect in goroutine 56
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/balancer_wrapper.go:300 +0x56
goroutine 1 gp=0xc0000061c0 m=nil [select, locked to thread]:
runtime.gopark(0xc0006c02e8?, 0x2?, 0x90?, 0x1?, 0xc0006c0254?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc0006c00f8 sp=0xc0006c00d8 pc=0x213e62e
runtime.selectgo(0xc0006c02e8, 0xc0006c0250, 0x1c0?, 0x0, 0xd27f10?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc0006c0218 sp=0xc0006c00f8 pc=0x21504a5
github.com/docker/docker/libcontainerd/supervisor.Start({0xd5bd58, 0xc000114fa0}, {0xc0005a05a0?, 0x22461b5?}, {0xc0005a05c0, 0x1a}, {0xc000175460, 0x3, 0x3fba2c0?})
	/go/src/github.com/docker/docker/libcontainerd/supervisor/remote_daemon.go:104 +0x5a5 fp=0xc0006c03c0 sp=0xc0006c0218 pc=0x3d93685
main.(*DaemonCli).initContainerd(0xc0005e23c0, {0xd5bd58, 0xc000114fa0})
	/go/src/github.com/docker/docker/cmd/dockerd/daemon_unix.go:141 +0x24e fp=0xc0006c04a0 sp=0xc0006c03c0 pc=0x3da184e
main.(*DaemonCli).start(0xc0005e23c0, 0xc000324f00)
	/go/src/github.com/docker/docker/cmd/dockerd/daemon.go:185 +0x8c9 fp=0xc0006c1c40 sp=0xc0006c04a0 pc=0x3d99149
main.runDaemon(...)
	/go/src/github.com/docker/docker/cmd/dockerd/docker_unix.go:13
main.newDaemonCommand.func1(0xc0005ee200?, {0xc0005ea840?, 0x7?, 0x8fed1b?})
	/go/src/github.com/docker/docker/cmd/dockerd/docker.go:40 +0x94 fp=0xc0006c1c70 sp=0xc0006c1c40 pc=0x3da1d74
github.com/spf13/cobra.(*Command).execute(0xc000322308, {0xc000110050, 0x3, 0x3})
	/go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:985 +0xaca fp=0xc0006c1df8 sp=0xc0006c1c70 pc=0x22f990a
github.com/spf13/cobra.(*Command).ExecuteC(0xc000322308)
	/go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:1117 +0x3ff fp=0xc0006c1ed0 sp=0xc0006c1df8 pc=0x22fa1df
github.com/spf13/cobra.(*Command).Execute(...)
	/go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:1041
main.main()
	/go/src/github.com/docker/docker/cmd/dockerd/docker.go:115 +0x1a6 fp=0xc0006c1f50 sp=0xc0006c1ed0 pc=0x3da1fe6
runtime.main()
	/usr/local/go/src/runtime/proc.go:271 +0x29d fp=0xc0006c1fe0 sp=0xc0006c1f50 pc=0x213e1dd
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0006c1fe8 sp=0xc0006c1fe0 pc=0x2176d41
goroutine 2 gp=0xc000006c40 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00008cfa8 sp=0xc00008cf88 pc=0x213e62e
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:408
runtime.forcegchelper()
	/usr/local/go/src/runtime/proc.go:326 +0xb3 fp=0xc00008cfe0 sp=0xc00008cfa8 pc=0x213e493
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00008cfe8 sp=0xc00008cfe0 pc=0x2176d41
created by runtime.init.6 in goroutine 1
	/usr/local/go/src/runtime/proc.go:314 +0x1a
goroutine 18 gp=0xc000102380 m=nil [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000088780 sp=0xc000088760 pc=0x213e62e
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:408
runtime.bgsweep(0xc0000b6000)
	/usr/local/go/src/runtime/mgcsweep.go:318 +0xdf fp=0xc0000887c8 sp=0xc000088780 pc=0x2127d5f
runtime.gcenable.gowrap1()
	/usr/local/go/src/runtime/mgc.go:203 +0x25 fp=0xc0000887e0 sp=0xc0000887c8 pc=0x211c665
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000887e8 sp=0xc0000887e0 pc=0x2176d41
created by runtime.gcenable in goroutine 1
	/usr/local/go/src/runtime/mgc.go:203 +0x66
goroutine 19 gp=0xc000102540 m=nil [GC scavenge wait]:
runtime.gopark(0x10000?, 0xd1e628?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000088f78 sp=0xc000088f58 pc=0x213e62e
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:408
runtime.(*scavengerState).park(0x3fb6ee0)
	/usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000088fa8 sp=0xc000088f78 pc=0x2125709
runtime.bgscavenge(0xc0000b6000)
	/usr/local/go/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc000088fc8 sp=0xc000088fa8 pc=0x2125cb9
runtime.gcenable.gowrap2()
	/usr/local/go/src/runtime/mgc.go:204 +0x25 fp=0xc000088fe0 sp=0xc000088fc8 pc=0x211c605
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000088fe8 sp=0xc000088fe0 pc=0x2176d41
created by runtime.gcenable in goroutine 1
	/usr/local/go/src/runtime/mgc.go:204 +0xa5
goroutine 20 gp=0xc000102a80 m=nil [finalizer wait]:
runtime.gopark(0xc00008c648?, 0x210eec5?, 0xa8?, 0x1?, 0xc0000061c0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00008c620 sp=0xc00008c600 pc=0x213e62e
runtime.runfinq()
	/usr/local/go/src/runtime/mfinal.go:194 +0x107 fp=0xc00008c7e0 sp=0xc00008c620 pc=0x211b627
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00008c7e8 sp=0xc00008c7e0 pc=0x2176d41
created by runtime.createfing in goroutine 1
	/usr/local/go/src/runtime/mfinal.go:164 +0x3d
goroutine 21 gp=0xc00028ee00 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000089750 sp=0xc000089730 pc=0x213e62e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc0000897e0 sp=0xc000089750 pc=0x211e745
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000897e8 sp=0xc0000897e0 pc=0x2176d41
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x1c
goroutine 22 gp=0xc00028efc0 m=nil [GC worker (idle)]:
runtime.gopark(0x170174425e1?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000089f50 sp=0xc000089f30 pc=0x213e62e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc000089fe0 sp=0xc000089f50 pc=0x211e745
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000089fe8 sp=0xc000089fe0 pc=0x2176d41
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x1c
goroutine 23 gp=0xc00028f180 m=nil [GC worker (idle)]:
runtime.gopark(0x17017442334?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00008a750 sp=0xc00008a730 pc=0x213e62e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc00008a7e0 sp=0xc00008a750 pc=0x211e745
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00008a7e8 sp=0xc00008a7e0 pc=0x2176d41
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x1c
goroutine 24 gp=0xc00028f340 m=nil [GC worker (idle)]:
runtime.gopark(0x170174404b8?, 0x3?, 0xee?, 0x8e?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00008af50 sp=0xc00008af30 pc=0x213e62e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc00008afe0 sp=0xc00008af50 pc=0x211e745
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00008afe8 sp=0xc00008afe0 pc=0x2176d41
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x1c
goroutine 25 gp=0xc00028f500 m=nil [GC worker (idle)]:
runtime.gopark(0x1701744027c?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00008b750 sp=0xc00008b730 pc=0x213e62e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc00008b7e0 sp=0xc00008b750 pc=0x211e745
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00008b7e8 sp=0xc00008b7e0 pc=0x2176d41
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x1c
goroutine 26 gp=0xc00028f6c0 m=nil [GC worker (idle)]:
runtime.gopark(0x1701744071a?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00008bf50 sp=0xc00008bf30 pc=0x213e62e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc00008bfe0 sp=0xc00008bf50 pc=0x211e745
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00008bfe8 sp=0xc00008bfe0 pc=0x2176d41
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x1c
goroutine 27 gp=0xc00028f880 m=nil [GC worker (idle)]:
runtime.gopark(0x4025000?, 0x1?, 0xbd?, 0xb6?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc0003ce750 sp=0xc0003ce730 pc=0x213e62e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc0003ce7e0 sp=0xc0003ce750 pc=0x211e745
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0003ce7e8 sp=0xc0003ce7e0 pc=0x2176d41
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x1c
goroutine 28 gp=0xc00028fa40 m=nil [GC worker (idle)]:
runtime.gopark(0x17017447284?, 0x0?, 0x0?, 0x0?, 0xc0003cf1e8?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc0003cef50 sp=0xc0003cef30 pc=0x213e62e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc0003cefe0 sp=0xc0003cef50 pc=0x211e745
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0003cefe8 sp=0xc0003cefe0 pc=0x2176d41
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x1c
goroutine 3 gp=0xc0005028c0 m=nil [select]:
runtime.gopark(0xc0003d1778?, 0x3?, 0xb8?, 0x75?, 0xc0003d1772?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc0003d1618 sp=0xc0003d15f8 pc=0x213e62e
runtime.selectgo(0xc0003d1778, 0xc0003d176c, 0xc000324a80?, 0x0, 0x0?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc0003d1738 sp=0xc0003d1618 pc=0x21504a5
go.opencensus.io/stats/view.(*worker).start(0xc000324a80)
	/go/src/github.com/docker/docker/vendor/go.opencensus.io/stats/view/worker.go:292 +0x9f fp=0xc0003d17c8 sp=0xc0003d1738 pc=0x3acc11f
go.opencensus.io/stats/view.init.0.gowrap1()
	/go/src/github.com/docker/docker/vendor/go.opencensus.io/stats/view/worker.go:34 +0x25 fp=0xc0003d17e0 sp=0xc0003d17c8 pc=0x3acb485
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0003d17e8 sp=0xc0003d17e0 pc=0x2176d41
created by go.opencensus.io/stats/view.init.0 in goroutine 1
	/go/src/github.com/docker/docker/vendor/go.opencensus.io/stats/view/worker.go:34 +0x8d
goroutine 52 gp=0xc0005d4380 m=nil [select]:
runtime.gopark(0xc0003cbd90?, 0x2?, 0xa0?, 0xbe?, 0xc0003cbd64?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00052dc10 sp=0xc00052dbf0 pc=0x213e62e
runtime.selectgo(0xc00052dd90, 0xc0003cbd60, 0x200000000?, 0x0, 0xc0003cbdb8?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc00052dd30 sp=0xc00052dc10 pc=0x21504a5
io.(*pipe).read(0xc0005cede0, {0xc0006da000, 0x10000, 0x210e31e?})
	/usr/local/go/src/io/pipe.go:57 +0xa5 fp=0xc00052ddc0 sp=0xc00052dd30 pc=0x21c40a5
io.(*PipeReader).Read(0xc0003cbe08?, {0xc0006da000?, 0x4025000?, 0xc0003cbe90?})
	/usr/local/go/src/io/pipe.go:134 +0x1a fp=0xc00052ddf0 sp=0xc00052ddc0 pc=0x21c47fa
bufio.(*Scanner).Scan(0xc00052df28)
	/usr/local/go/src/bufio/scan.go:219 +0x81e fp=0xc00052dec8 sp=0xc00052ddf0 pc=0x228c35e
github.com/sirupsen/logrus.(*Entry).writerScanner(0xc000396e00, 0xc0005cede0, 0xc000584c50)
	/go/src/github.com/docker/docker/vendor/github.com/sirupsen/logrus/writer.go:86 +0x11d fp=0xc00052dfb8 sp=0xc00052dec8 pc=0x236ddbd
github.com/sirupsen/logrus.(*Entry).WriterLevel.gowrap1()
	/go/src/github.com/docker/docker/vendor/github.com/sirupsen/logrus/writer.go:57 +0x28 fp=0xc00052dfe0 sp=0xc00052dfb8 pc=0x236dc68
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00052dfe8 sp=0xc00052dfe0 pc=0x2176d41
created by github.com/sirupsen/logrus.(*Entry).WriterLevel in goroutine 1
	/go/src/github.com/docker/docker/vendor/github.com/sirupsen/logrus/writer.go:57 +0x31f
goroutine 51 gp=0xc000603c00 m=nil [select]:
runtime.gopark(0xc000531d90?, 0x2?, 0x38?, 0x1c?, 0xc000531d64?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000531c10 sp=0xc000531bf0 pc=0x213e62e
runtime.selectgo(0xc000531d90, 0xc000531d60, 0x0?, 0x0, 0xc0003d1db8?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000531d30 sp=0xc000531c10 pc=0x21504a5
io.(*pipe).read(0xc0005ced80, {0xc00051a1b5, 0xfe4b, 0x0?})
	/usr/local/go/src/io/pipe.go:57 +0xa5 fp=0xc000531dc0 sp=0xc000531d30 pc=0x21c40a5
io.(*PipeReader).Read(0xc00051a000?, {0xc00051a1b5?, 0xc000531e88?, 0xc000531e90?})
	/usr/local/go/src/io/pipe.go:134 +0x1a fp=0xc000531df0 sp=0xc000531dc0 pc=0x21c47fa
bufio.(*Scanner).Scan(0xc000531f28)
	/usr/local/go/src/bufio/scan.go:219 +0x81e fp=0xc000531ec8 sp=0xc000531df0 pc=0x228c35e
github.com/sirupsen/logrus.(*Entry).writerScanner(0xc000396e00, 0xc0005ced80, 0xc000584c40)
	/go/src/github.com/docker/docker/vendor/github.com/sirupsen/logrus/writer.go:86 +0x11d fp=0xc000531fb8 sp=0xc000531ec8 pc=0x236ddbd
github.com/sirupsen/logrus.(*Entry).WriterLevel.gowrap1()
	/go/src/github.com/docker/docker/vendor/github.com/sirupsen/logrus/writer.go:57 +0x28 fp=0xc000531fe0 sp=0xc000531fb8 pc=0x236dc68
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000531fe8 sp=0xc000531fe0 pc=0x2176d41
created by github.com/sirupsen/logrus.(*Entry).WriterLevel in goroutine 1
	/go/src/github.com/docker/docker/vendor/github.com/sirupsen/logrus/writer.go:57 +0x31f
goroutine 66 gp=0xc000007a40 m=8 mp=0xc00023b808 [syscall, locked to thread]:
syscall.Syscall6(0xf7, 0x1, 0xa1, 0xc00079be08, 0x1000004, 0x0, 0x0)
	/usr/local/go/src/syscall/syscall_linux.go:91 +0x39 fp=0xc00079bdd0 sp=0xc00079bd70 pc=0x21d0ff9
os.(*Process).blockUntilWaitable(0xc00072c120)
	/usr/local/go/src/os/wait_waitid.go:32 +0x76 fp=0xc00079bea8 sp=0xc00079bdd0 pc=0x21ff9b6
os.(*Process).wait(0xc00072c120)
	/usr/local/go/src/os/exec_unix.go:22 +0x25 fp=0xc00079bf08 sp=0xc00079bea8 pc=0x21f8cc5
os.(*Process).Wait(...)
	/usr/local/go/src/os/exec.go:134
os/exec.(*Cmd).Wait(0xc0001f2f00)
	/usr/local/go/src/os/exec/exec.go:906 +0x45 fp=0xc00079bf68 sp=0xc00079bf08 pc=0x27052c5
github.com/docker/docker/libcontainerd/supervisor.(*remote).startContainerd.func1()
	/go/src/github.com/docker/docker/libcontainerd/supervisor/remote_daemon.go:201 +0xbc fp=0xc00079bfe0 sp=0xc00079bf68 pc=0x3d9477c
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00079bfe8 sp=0xc00079bfe0 pc=0x2176d41
created by github.com/docker/docker/libcontainerd/supervisor.(*remote).startContainerd in goroutine 53
	/go/src/github.com/docker/docker/libcontainerd/supervisor/remote_daemon.go:182 +0x51b
goroutine 50 gp=0xc000007c00 m=nil [select]:
runtime.gopark(0xc00009fd90?, 0x2?, 0x38?, 0xfc?, 0xc00009fd64?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00009fc10 sp=0xc00009fbf0 pc=0x213e62e
runtime.selectgo(0xc00009fd90, 0xc00009fd60, 0x200000000?, 0x0, 0xc00008ddb8?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc00009fd30 sp=0xc00009fc10 pc=0x21504a5
io.(*pipe).read(0xc0005ced20, {0xc0006ca970, 0xf690, 0x210e200?})
	/usr/local/go/src/io/pipe.go:57 +0xa5 fp=0xc00009fdc0 sp=0xc00009fd30 pc=0x21c40a5
io.(*PipeReader).Read(0xc0006ca7bb?, {0xc0006ca970?, 0xc00009fe88?, 0xc00009fe90?})
	/usr/local/go/src/io/pipe.go:134 +0x1a fp=0xc00009fdf0 sp=0xc00009fdc0 pc=0x21c47fa
bufio.(*Scanner).Scan(0xc00009ff28)
	/usr/local/go/src/bufio/scan.go:219 +0x81e fp=0xc00009fec8 sp=0xc00009fdf0 pc=0x228c35e
github.com/sirupsen/logrus.(*Entry).writerScanner(0xc000396e00, 0xc0005ced20, 0xc000584c30)
	/go/src/github.com/docker/docker/vendor/github.com/sirupsen/logrus/writer.go:86 +0x11d fp=0xc00009ffb8 sp=0xc00009fec8 pc=0x236ddbd
github.com/sirupsen/logrus.(*Entry).WriterLevel.gowrap1()
	/go/src/github.com/docker/docker/vendor/github.com/sirupsen/logrus/writer.go:57 +0x28 fp=0xc00009ffe0 sp=0xc00009ffb8 pc=0x236dc68
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00009ffe8 sp=0xc00009ffe0 pc=0x2176d41
created by github.com/sirupsen/logrus.(*Entry).WriterLevel in goroutine 1
	/go/src/github.com/docker/docker/vendor/github.com/sirupsen/logrus/writer.go:57 +0x31f
goroutine 53 gp=0xc0005a2e00 m=nil [select]:
runtime.gopark(0xc00070f630?, 0x2?, 0x8?, 0x0?, 0xc00070f584?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00070f430 sp=0xc00070f410 pc=0x213e62e
runtime.selectgo(0xc00070f630, 0xc00070f580, 0x5eb980?, 0x0, 0xd0?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc00070f550 sp=0xc00070f430 pc=0x21504a5
google.golang.org/grpc.(*pickerWrapper).pick(0xc0001756a0, {0xd5bd20, 0xc0005eb980}, 0x0, {{0x962ad6?, 0xc0005eb980?}, {0xd5bd20?, 0xc0005eb980?}})
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/picker_wrapper.go:120 +0x166 fp=0xc00070f6a0 sp=0xc00070f550 pc=0x298b446
google.golang.org/grpc.(*ClientConn).getTransport(0x0?, {0xd5bd20?, 0xc0005eb980?}, 0x1b?, {0x962ad6?, 0x210c2bf?})
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:1050 +0x2f fp=0xc00070f6f0 sp=0xc00070f6a0 pc=0x2985e0f
google.golang.org/grpc.(*csAttempt).getTransport(0xc0005f1e10)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:468 +0x45 fp=0xc00070f730 sp=0xc00070f6f0 pc=0x299fa85
google.golang.org/grpc.newClientStreamWithParams.func2(0xc0005f1e10)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:351 +0x25 fp=0xc00070f750 sp=0xc00070f730 pc=0x299f065
google.golang.org/grpc.(*clientStream).withRetry(0xc0005bb680, 0xc000585e50, 0xc00070f8e0)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:789 +0x13a fp=0xc00070f7c0 sp=0xc00070f750 pc=0x29a0cda
google.golang.org/grpc.newClientStreamWithParams({0xd5bd90, 0xc000397a40}, 0x3fb44a0, 0xc0005a7008, {0x962ad6, 0x1c}, {0x0, 0x0, 0x0, 0x0, ...}, ...)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:363 +0xb9d fp=0xc00070f938 sp=0xc00070f7c0 pc=0x299eb1d
google.golang.org/grpc.newClientStream.func3({0xd5bd90?, 0xc000397a40?}, 0xc000397a40?)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:220 +0x87 fp=0xc00070f9c8 sp=0xc00070f938 pc=0x299dec7
google.golang.org/grpc.newClientStream({0xd5bd90, 0xc000397a40}, 0x3fb44a0, 0xc0005a7008, {0x962ad6, 0x1c}, {0xc0007927e0, 0x6, 0x60?})
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:255 +0x783 fp=0xc00070fb30 sp=0xc00070f9c8 pc=0x299d8a3
google.golang.org/grpc.invoke({0xd5bd90?, 0xc000397a40?}, {0x962ad6?, 0x53e320?}, {0x66cce0, 0xc0005e3140}, {0x66ce60, 0xc0005eb890}, 0x10?, {0xc0007927e0, ...})
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/call.go:66 +0x77 fp=0xc00070fb98 sp=0xc00070fb30 pc=0x2980517
github.com/moby/buildkit/util/grpcerrors.UnaryClientInterceptor({0xd5bd90?, 0xc000397a40?}, {0x962ad6?, 0xc000600008?}, {0x66cce0?, 0xc0005e3140?}, {0x66ce60?, 0xc0005eb890?}, 0xc00070fca0?, 0xa3d538, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/grpcerrors/intercept.go:41 +0x7e fp=0xc00070fc18 sp=0xc00070fb98 pc=0x2effb1e
google.golang.org/grpc.(*ClientConn).Invoke(0xc0005a7008, {0xd5bd90?, 0xc000397a40?}, {0x962ad6?, 0x1c?}, {0x66cce0?, 0xc0005e3140?}, {0x66ce60?, 0xc0005eb890?}, {0xc000175a00, ...})
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/call.go:35 +0x205 fp=0xc00070fcb0 sp=0xc00070fc18 pc=0x29803e5
google.golang.org/grpc/health/grpc_health_v1.(*healthClient).Check(0xc000585df0, {0xd5bd90, 0xc000397a40}, 0xc0005e3140, {0xc000585e00, 0x1, 0xd5bd58?})
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/health/grpc_health_v1/health_grpc.pb.go:90 +0x167 fp=0xc00070fd40 sp=0xc00070fcb0 pc=0x2e1a5c7
github.com/containerd/containerd.(*Client).IsServing(0xc0005ee500, {0xd5bd90, 0xc000397a40})
	/go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/client.go:259 +0x122 fp=0xc00070fdb8 sp=0xc00070fd40 pc=0x2eafb82
github.com/docker/docker/libcontainerd/supervisor.(*remote).monitorDaemon(0xc000586488, {0xd5bd58, 0xc000114fa0})
	/go/src/github.com/docker/docker/libcontainerd/supervisor/remote_daemon.go:322 +0x8df fp=0xc00070ffb8 sp=0xc00070fdb8 pc=0x3d9517f
github.com/docker/docker/libcontainerd/supervisor.Start.gowrap1()
	/go/src/github.com/docker/docker/libcontainerd/supervisor/remote_daemon.go:99 +0x28 fp=0xc00070ffe0 sp=0xc00070ffb8 pc=0x3d93948
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00070ffe8 sp=0xc00070ffe0 pc=0x2176d41
created by github.com/docker/docker/libcontainerd/supervisor.Start in goroutine 1
	/go/src/github.com/docker/docker/libcontainerd/supervisor/remote_daemon.go:99 +0x4ef
goroutine 54 gp=0xc0005a2fc0 m=nil [select]:
runtime.gopark(0xc0003cc760?, 0x2?, 0x0?, 0x0?, 0xc0003cc724?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc0003cc5c8 sp=0xc0003cc5a8 pc=0x213e62e
runtime.selectgo(0xc0003cc760, 0xc0003cc720, 0xc0002b9140?, 0x0, 0xc0003cc740?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc0003cc6e8 sp=0xc0003cc5c8 pc=0x21504a5
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc0005853c0, {0xd5bd58, 0xc000115180})
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0x115 fp=0xc0003cc7b8 sp=0xc0003cc6e8 pc=0x28cf995
google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer.gowrap1()
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x28 fp=0xc0003cc7e0 sp=0xc0003cc7b8 pc=0x28cf768
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0003cc7e8 sp=0xc0003cc7e0 pc=0x2176d41
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 53
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x11a
goroutine 55 gp=0xc0005a3180 m=nil [select]:
runtime.gopark(0xc0007a5f60?, 0x2?, 0xe8?, 0xc4?, 0xc0007a5f24?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc0007a5dc8 sp=0xc0007a5da8 pc=0x213e62e
runtime.selectgo(0xc0007a5f60, 0xc0007a5f20, 0xc0003df8d0?, 0x0, 0x0?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc0007a5ee8 sp=0xc0007a5dc8 pc=0x21504a5
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc0005853f0, {0xd5bd58, 0xc0001151d0})
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0x115 fp=0xc0007a5fb8 sp=0xc0007a5ee8 pc=0x28cf995
google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer.gowrap1()
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x28 fp=0xc0007a5fe0 sp=0xc0007a5fb8 pc=0x28cf768
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0007a5fe8 sp=0xc0007a5fe0 pc=0x2176d41
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 53
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x11a
goroutine 56 gp=0xc0005a3340 m=nil [select]:
runtime.gopark(0xc00070bf60?, 0x2?, 0x0?, 0x0?, 0xc00070bf24?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00070bdc8 sp=0xc00070bda8 pc=0x213e62e
runtime.selectgo(0xc00070bf60, 0xc00070bf20, 0x0?, 0x0, 0x0?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc00070bee8 sp=0xc00070bdc8 pc=0x21504a5
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc000585420, {0xd5bd58, 0xc000115220})
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0x115 fp=0xc00070bfb8 sp=0xc00070bee8 pc=0x28cf995
google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer.gowrap1()
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x28 fp=0xc00070bfe0 sp=0xc00070bfb8 pc=0x28cf768
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00070bfe8 sp=0xc00070bfe0 pc=0x2176d41
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 53
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x11a
Reproduce

Start docker with the following command line

dockerd --host=unix:///var/run/dind/docker.sock

Host machine:

Linux[REDACTED].ec2.internal 6.1.115 #1 SMP PREEMPT_DYNAMIC Fri Nov 15 19:15:26 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

AMI ID: bottlerocket-aws-k8s-1.31-x86_64-v1.27.1-efd46c32

Host OS:

amazon/bottlerocket-aws-k8s-1.31-x86_64-v1.27.1-efd46c32

It is even stranger that somes nodes are able to start dockerd normally, but some don't. We couldn't diagnose any further. Tried to check with killsnoop, but nothing strange is happening. dmesg is also clean on both container and host.

containerd is able to start normally.

Notice: the container running dind has privileged: true on its spec.

Expected behavior

dockerd should start normally. Instead, it immediately crashes.

docker version
Client:
 Version:    27.5.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.19.3
    Path:     /usr/local/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  083f676
    Path:     /usr/local/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 4
  Running: 0
  Paused: 0
  Stopped: 4
 Images: 1
 Server Version: 27.5.0
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
 runc version: v1.2.4-0-g6c52b3f
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.1.115
 Operating System: Alpine Linux v3.21 (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 20.02GiB
 Name: [REDACTED]-68f5497c89-qdzgh
 ID: 5b1e2919-14a0-4eb4-9819-e733dba36e5b
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Registry Mirrors:
  https://[REDACTED]/docker/
 Live Restore Enabled: false
 Product License: Community Engine
docker info
docker version
Client:
 Version:           27.5.0
 API version:       1.47
 Go version:        go1.22.10
 Git commit:        a187fa5
 Built:             Mon Jan 13 15:23:50 2025
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          27.5.0
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.10
  Git commit:       38b84dc
  Built:            Mon Jan 13 15:25:13 2025
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.25
  GitCommit:        bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
 runc:
  Version:          1.2.4
  GitCommit:        v1.2.4-0-g6c52b3f
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
Additional Info

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the crash when starting dockerd in dind, then follow initContainerd in cmd/dockerd/daemon_unix.go through libcontainerd/supervisor/remote_daemon.go. Inspect the reported call path in vendor/google.golang.org/grpc/clientconn.go and compare the runtime state at resolveNow; done means dockerd starts without the SIGSEGV in the reported environment.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go, grpc
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.