docker daemon occasionally returns error: failed to set IPv6 gateway while updating gateway: file exists
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 2.2k
- Forks
- 875
- PR merge metrics
- No merged PRs in 30d
Description
I have a python application that runs on a docker host (Ubuntu). The application basically allows users to concurrently start multiple containers on IPv4/IPv6 dualstack Macvlan network.
For example, if a user requests 3 containers, then the application tries to create and start 3 containers concurrently using 3 threads.
thread1: create -> start
thread2: create -> start
thread3: create -> start
Each container gets both IPv4 and IPv6 addresses assigned from the Macvlan network's IPv4/IPv6 pools.
Issue
The issue I encountered is the docker daemon occasionally returns below error when starting containers, and one of containers sometimes fails to start. It took about 250 times of attempts (each attempt = starting 2 containers concurrently and delete them) to see the error for 10 times.
docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused "process_linux.go:413: running prestart hook 0 caused \"error running hook: exit status 1, stdout: , stderr: time=\\\"2019-09-25T16:14:12-07:00\\\" level=fatal msg=\\\"failed to set IPv6 gateway while updating gateway: file exists\\\"\\n\""": unknown.
The issue does not happen when IPv6 is not enabled, or when IPv6 is enabled but starting just one container at a time.
This makes me assume that there is an issue where IPv6 network is not properly handled for concurrent access.
System information
lsb_release -a
Distributor ID: Ubuntu
Description: Ubuntu 18.04.2 LTS
Release: 18.04
Codename: bionic
uname -a
Linux hostname 4.4.0-159-generic #187-Ubuntu SMP Thu Aug 1 16:28:06 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
docker version
Client: Docker Engine - Community
Version: 19.03.2
API version: 1.40
Go version: go1.12.8
Git commit: 6a30dfc
Built: Thu Aug 29 05:28:19 2019
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.2
API version: 1.40 (minimum version 1.12)
Go version: go1.12.8
Git commit: 6a30dfc
Built: Thu Aug 29 05:26:54 2019
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.2.6
GitCommit: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
runc:
Version: 1.0.0-rc8
GitCommit: 425e105d5a03fabd737a126ad93d62a9eeede87f
docker-init:
Version: 0.18.0
GitCommit: fec3683
Note
I've seen this issue on Ubuntu 16.04, docker version 17.09.0-ce too.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing concurrent create-and-start operations for dual-stack IPv4/IPv6 Macvlan containers on the listed Ubuntu and Docker versions. Trace the daemon's IPv6 gateway setup and the prestart hook associated with the reported error. Done means concurrent container starts no longer intermittently fail with “failed to set IPv6 gateway while updating gateway: file exists”.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go, linux
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100