Docker containers managed by Nomad in bridge network mode are brought back up with broken networks.
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
```
Nomad v1.7.4
BuildDate 2024-02-08T14:34:12Z
Revision 29019121564e2ef7f5e2a227af6b959510bcc142
```
Though we are hitting it in v1.7.2 as well
### Operating system and Environment details
```
root@client-1:~# uname -a
Linux client-1 5.15.0-67-generic #74-Ubuntu SMP Wed Feb 22 14:14:39 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
root@client-1:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.2 LTS
Release: 22.04
Codename: jammy
```
We have hit this on multiple machines with slightly different versions, though all are Ubuntu 22.04. These are the details of a completely fresh Digital Ocean instance I used to reproduce the bug.
### Issue
We have noticed that when we restart the Docker daemon on our machines every Nomad job on the client is brought back up with a busted network. To be more specific, it is brought up with no network. For example, my test container before restarting docker has the following networks:
```
$ ip addr
1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0@if8: mtu 1500 qdisc noqueue state UP group default
link/ether 82:4c:5d:70:4e:fc brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.26.64.4/20 brd 172.26.79.255 scope global eth0
valid_lft forever preferred_lft foreve
```
and after restarting the daemon, is brought back up with just loopback:
```
$ ip addr
1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft foreve
```
This happens with every container, including the Nomad init container. Docker restarts the containers (as expected), the veths get recreated (as expected), but the containers now lack any interfaces other than loopback (unexpected).
Things that might be notable, the `nomad` network changes from `` to `` and on machines with `systemd-networkd`, it's logs complain about the veth's loosing carrier.
### Reproduction steps
1. Spin up a fresh Ubuntu 22.04 server (I used a Digital Ocean droplet for our reproduction but we've noticed this happening across our fleet so I don't think they're doing anything weird).
2. Install `docker-ce` as per [their docs](https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository) (I used Docker's apt registry to install it).
3. Install Nomad as per [the docs](https://developer.hashicorp.com/nomad/docs/install) (for the reproduction I specifically used the version of Nomad from Hashicorps repos).
4. Install the base CNI plugins by placing the contents of `https://github.com/containernetworking/plugins/releases/download/v1.0.0/cni-plugins-linux-amd64-v1.0.0.tgz`into `/opt/cni/bin`
5. `systemctl start docker`
6. `systemctl start nomad`
7. Run literally any job (I've included my job file below but we've seen this happen with many jobs)
8. `systemctl restart docker`
#### Expected Result
The ip/port combo that the job binds should be `curl`-able. It is before docker is restarted.
#### Actual Result
If you curl the ip/port combo it will complain about having no route to host:
```
root@client-1:~# curl -v localhost:27846
* Trying 127.0.0.1:27846...
* Trying ::1:27846...
* connect to ::1 port 27846 failed: Connection refused
* connect to 127.0.0.1 port 27846 failed: No route to host
* Failed to connect to localhost port 27846 after 3061 ms: No route to host
* Closing connection 0
curl: (7) Failed to connect to localhost port 27846 after 3061 ms: No route to host
```
This makes sense as executing `ip addr` from within the container will now reveal the container has lost it's bridge network veth.
### Job file (if appropriate)
We've noticed this happen with every job but the job file I used for the reproduction is:
```
job "jess-test-job" {
type = "service"
datacenters = ["*"]
group "http" {
network {
mode = "bridge"
port "http" {
to = "80"
}
}
task "whoami" {
driver = "docker"
config {
image = "strm/helloworld-http"
ports = ["http"]
}
}
}
}
```
The toy instance I used for reproduction has a broken journal so sadly I have no logs from that to provide. If reproduction turns out to be an issue I'd be happy to send over some logs from one of our actual failing instances but I have a hunch this won't be that hard to reproduce.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the failure using the Ubuntu 22.04, Docker, Nomad, CNI plugin, and bridge-mode job steps in the report, then inspect how Nomad and Docker handle container networking after a daemon restart. Done means restarted containers retain their bridge interfaces and the job's advertised IP and port remain reachable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go
- Domain
- devops, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100