Failure during overlay endpoints restore: restore network sandbox failed: network sandbox join failed: could not get network sandbox (oper true): failed get network namespace \"\": no such file or directory
Open
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 2.2k
- Forks
- 875
- PR merge metrics
- No merged PRs in 30d
Description
i use consul create overlay network,when i reboot slaver docker,the container con't boot.the log is:
an 10 17:27:38 docker systemd: Stopped Docker Application Container Engine.
Jan 10 17:27:38 docker systemd: Closed Docker Socket for the API.
Jan 10 17:28:35 docker systemd: Starting Docker Socket for the API.
Jan 10 17:28:35 docker systemd: Listening on Docker Socket for the API.
Jan 10 17:28:35 docker systemd: Starting Docker Application Container Engine...
Jan 10 17:28:35 docker dockerd: time="2019-01-10T17:28:35.851796952+08:00" level=warning msg="[!] DON'T BIND ON ANY IP ADDRESS WITHOUT setting --tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOING [!]"
Jan 10 17:28:35 docker dockerd: time="2019-01-10T17:28:35.857206117+08:00" level=info msg="parsed scheme: \"unix\"" module=grpc
Jan 10 17:28:35 docker dockerd: time="2019-01-10T17:28:35.857232538+08:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Jan 10 17:28:35 docker dockerd: time="2019-01-10T17:28:35.857298978+08:00" level=info msg="parsed scheme: \"unix\"" module=grpc
Jan 10 17:28:35 docker dockerd: time="2019-01-10T17:28:35.857310322+08:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Jan 10 17:28:35 docker dockerd: time="2019-01-10T17:28:35.857410658+08:00" level=info msg="ccResolverWrapper: sending new addresses to cc: [{unix:///run/containerd/containerd.sock 0 <nil>}]" module=grpc
Jan 10 17:28:35 docker dockerd: time="2019-01-10T17:28:35.857437934+08:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Jan 10 17:28:35 docker dockerd: time="2019-01-10T17:28:35.857432691+08:00" level=info msg="ccResolverWrapper: sending new addresses to cc: [{unix:///run/containerd/containerd.sock 0 <nil>}]" module=grpc
Jan 10 17:28:35 docker dockerd: time="2019-01-10T17:28:35.857481142+08:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Jan 10 17:28:35 docker dockerd: time="2019-01-10T17:28:35.857542797+08:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc4201a5d50, CONNECTING" module=grpc
Jan 10 17:28:35 docker dockerd: time="2019-01-10T17:28:35.857548281+08:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420866390, CONNECTING" module=grpc
Jan 10 17:28:35 docker dockerd: time="2019-01-10T17:28:35.857808502+08:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc4201a5d50, READY" module=grpc
Jan 10 17:28:35 docker dockerd: time="2019-01-10T17:28:35.859888418+08:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420866390, READY" module=grpc
Jan 10 17:28:35 docker dockerd: time="2019-01-10T17:28:35.869570799+08:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
Jan 10 17:28:35 docker dockerd: time="2019-01-10T17:28:35.892104466+08:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Jan 10 17:28:35 docker dockerd: time="2019-01-10T17:28:35.892138947+08:00" level=info msg="Initializing discovery without TLS"
Jan 10 17:28:35 docker dockerd: time="2019-01-10T17:28:35.893062919+08:00" level=info msg="Loading containers: start."
Jan 10 17:28:36 docker dockerd: time="2019-01-10T17:28:36.202705906+08:00" level=warning msg="Failure during overlay endpoints restore: restore network sandbox failed: network sandbox join failed: could not get network sandbox (oper true): failed get network namespace \"\": no such file or directory"
Jan 10 17:28:36 docker dockerd: time="2019-01-10T17:28:36.412281152+08:00" level=info msg="2019/01/10 17:28:36 [INFO] serf: EventMemberJoin: docker.huitu 10.0.0.214\n"
Jan 10 17:28:36 docker dockerd: time="2019-01-10T17:28:36.413599205+08:00" level=info msg="2019/01/10 17:28:36 [INFO] serf: EventMemberJoin: ywzx-h 10.0.0.213\n"
Jan 10 17:28:36 docker dockerd: time="2019-01-10T17:28:36.465316150+08:00" level=error msg="getNetworkFromStore for nid 40bef7593c70ff24c510ea511e773c4123381ba229ed5f3b2043e9aca470c281 failed while trying to build sandbox for cleanup: network 40bef7593c70ff24c510ea511e773c4123381ba229ed5f3b2043e9aca470c281 not found"
Jan 10 17:28:36 docker dockerd: time="2019-01-10T17:28:36.466916720+08:00" level=error msg="getEndpointFromStore for eid ad95c56226a60412db1382c4e17bbe07813ce0be502a63da9539a1246dfe0a20 failed while trying to build sandbox for cleanup: could not find endpoint ad95c56226a60412db1382c4e17bbe07813ce0be502a63da9539a1246dfe0a20: []"
Jan 10 17:28:36 docker dockerd: time="2019-01-10T17:28:36.466999665+08:00" level=info msg="Removing stale sandbox cf0585e050bfcca6a58795877fc6c616e28517fda0ce6997e0ea62ea689eb5ff (d4f740a7bc7bb44d60dc35d2e49bc185d81b46a8daad2d9c49509b5423bfc4df)"
Jan 10 17:28:36 docker dockerd: time="2019-01-10T17:28:36.467376450+08:00" level=warning msg="Failed getting network for ep 0112b3ab98b1d8e020f9d955dbe441a9647f603f934ef6b5e72d6b13c0a63779 during sandbox cf0585e050bfcca6a58795877fc6c616e28517fda0ce6997e0ea62ea689eb5ff delete: network 40bef7593c70ff24c510ea511e773c4123381ba229ed5f3b2043e9aca470c281 not found"
Jan 10 17:28:36 docker dockerd: time="2019-01-10T17:28:36.468508461+08:00" level=warning msg="Failed deleting endpoint ad95c56226a60412db1382c4e17bbe07813ce0be502a63da9539a1246dfe0a20: failed to get endpoint from store during Delete: could not find endpoint ad95c56226a60412db1382c4e17bbe07813ce0be502a63da9539a1246dfe0a20: []\n"
Jan 10 17:28:36 docker dockerd: time="2019-01-10T17:28:36.468556810+08:00" level=error msg="Failed to delete sandbox cf0585e050bfcca6a58795877fc6c616e28517fda0ce6997e0ea62ea689eb5ff while trying to cleanup: could not cleanup all the endpoints in container d4f740a7bc7bb44d60dc35d2e49bc185d81b46a8daad2d9c49509b5423bfc4df / sandbox cf0585e050bfcca6a58795877fc6c616e28517fda0ce6997e0ea62ea689eb5ff"
Jan 10 17:28:36 docker dockerd: time="2019-01-10T17:28:36.539422395+08:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Jan 10 17:28:36 docker dockerd: time="2019-01-10T17:28:36.754866274+08:00" level=error msg="0027a2ca3ae6c74aa64e98276d92f4929246b671578d4a456a5e1789d8861dda cleanup: failed to delete container from containerd: no such container"
Jan 10 17:28:36 docker dockerd: time="2019-01-10T17:28:36.754903084+08:00" level=error msg="Failed to start container 0027a2ca3ae6c74aa64e98276d92f4929246b671578d4a456a5e1789d8861dda: network sandbox join failed: network sandbox join failed: could not get network sandbox (oper true): failed get network namespace \"\": no such file or directory"
Jan 10 17:28:36 docker dockerd: time="2019-01-10T17:28:36.757931202+08:00" level=error msg="c158fdacf7b090efbc490ef797c4180997a769f79fdd5172e20c3309298ed718 cleanup: failed to delete container from containerd: no such container"
Jan 10 17:28:36 docker dockerd: time="2019-01-10T17:28:36.757960363+08:00" level=error msg="Failed to start container c158fdacf7b090efbc490ef797c4180997a769f79fdd5172e20c3309298ed718: network sandbox join failed: network sandbox join failed: could not get network sandbox (oper true): failed get network namespace \"\": no such file or directory"
Jan 10 17:28:36 docker dockerd: time="2019-01-10T17:28:36.793220555+08:00" level=error msg="7efe5e326effc3a1a2d54958b20a59f162d45260f8c3968391efc6614d4016dd cleanup: failed to delete container from containerd: no such container"
Jan 10 17:28:36 docker dockerd: time="2019-01-10T17:28:36.793262289+08:00" level=error msg="Failed to start container 7efe5e326effc3a1a2d54958b20a59f162d45260f8c3968391efc6614d4016dd: network sandbox join failed: network sandbox join failed: could not get network sandbox (oper true): failed get network namespace \"\": no such file or directory"
Jan 10 17:28:36 docker dockerd: time="2019-01-10T17:28:36.803230494+08:00" level=error msg="24cecebf52d2962cc388ab3dca47b3e6a9229a304f19e32a60c8404ffade64b9 cleanup: failed to delete container from containerd: no such container"
Jan 10 17:28:36 docker dockerd: time="2019-01-10T17:28:36.803259685+08:00" level=error msg="Failed to start container 24cecebf52d2962cc388ab3dca47b3e6a9229a304f19e32a60c8404ffade64b9: network sandbox join failed: network sandbox join failed: could not get network sandbox (oper true): failed get network namespace \"\": no such file or directory"
Jan 10 17:28:36 docker dockerd: time="2019-01-10T17:28:36.811000316+08:00" level=error msg="2aebb2cb08b7bcf1554c665d8425c2507ff76eb98a85dce5cfee4ffb9ef377aa cleanup: failed to delete container from containerd: no such container"
Jan 10 17:28:36 docker dockerd: time="2019-01-10T17:28:36.811033651+08:00" level=error msg="Failed to start container 2aebb2cb08b7bcf1554c665d8425c2507ff76eb98a85dce5cfee4ffb9ef377aa: network sandbox join failed: network sandbox join failed: could not get network sandbox (oper true): failed get network namespace \"\": no such file or directory"
Jan 10 17:28:36 docker dockerd: time="2019-01-10T17:28:36.811069988+08:00" level=info msg="Loading containers: done."
Jan 10 17:28:36 docker dockerd: time="2019-01-10T17:28:36.861053493+08:00" level=info msg="Docker daemon" commit=bca0068 graphdriver(s)=overlay2 version=18.09.1-rc1
Jan 10 17:28:36 docker dockerd: time="2019-01-10T17:28:36.861179167+08:00" level=info msg="Daemon has completed initialization"
Jan 10 17:28:36 docker dockerd: time="2019-01-10T17:28:36.864091530+08:00" level=warning msg="Could not register builder git source: failed to find git binary: exec: \"git\": executable file not found in $PATH"
Jan 10 17:28:36 docker dockerd: time="2019-01-10T17:28:36.870439152+08:00" level=info msg="API listen on [::]:2375"
Jan 10 17:28:36 docker dockerd: time="2019-01-10T17:28:36.870500757+08:00" level=info msg="API listen on /var/run/docker.sock"
Jan 10 17:28:36 docker systemd: Started Docker Application Container Engine.
Jan 10 17:28:36 docker dockerd: time="2019-01-10T17:28:36.871330446+08:00" level=info msg="API listen on /var/run/docker.sock"
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source files or tests are named. Start by reproducing the Docker daemon restart on the Consul-backed overlay network and trace the “Failure during overlay endpoints restore” and “network sandbox join failed” logs. Done means containers recover after reboot without missing network namespaces or stale overlay endpoints.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100