hashicorp / hashicorp/nomad

Docker containers managed by Nomad in bridge network mode are brought back up with broken networks.

Open
#19,962 8 comments 0 reactions 0 assignees View on GitHub
stage/accepted theme/driver/docker theme/networking type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
```
Nomad v1.7.4
BuildDate 2024-02-08T14:34:12Z
Revision 29019121564e2ef7f5e2a227af6b959510bcc142
```

Though we are hitting it in v1.7.2 as well

### Operating system and Environment details
```
root@client-1:~# uname -a
Linux client-1 5.15.0-67-generic #74-Ubuntu SMP Wed Feb 22 14:14:39 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
root@client-1:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.2 LTS
Release: 22.04
Codename: jammy
```

We have hit this on multiple machines with slightly different versions, though all are Ubuntu 22.04. These are the details of a completely fresh Digital Ocean instance I used to reproduce the bug.

### Issue

We have noticed that when we restart the Docker daemon on our machines every Nomad job on the client is brought back up with a busted network. To be more specific, it is brought up with no network. For example, my test container before restarting docker has the following networks:

```
$ ip addr
1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0@if8: mtu 1500 qdisc noqueue state UP group default
link/ether 82:4c:5d:70:4e:fc brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.26.64.4/20 brd 172.26.79.255 scope global eth0
valid_lft forever preferred_lft foreve
```

and after restarting the daemon, is brought back up with just loopback:
```
$ ip addr
1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft foreve
```

This happens with every container, including the Nomad init container. Docker restarts the containers (as expected), the veths get recreated (as expected), but the containers now lack any interfaces other than loopback (unexpected).

Things that might be notable, the `nomad` network changes from `` to `` and on machines with `systemd-networkd`, it's logs complain about the veth's loosing carrier.

### Reproduction steps

1. Spin up a fresh Ubuntu 22.04 server (I used a Digital Ocean droplet for our reproduction but we've noticed this happening across our fleet so I don't think they're doing anything weird).
2. Install `docker-ce` as per [their docs](https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository) (I used Docker's apt registry to install it).
3. Install Nomad as per [the docs](https://developer.hashicorp.com/nomad/docs/install) (for the reproduction I specifically used the version of Nomad from Hashicorps repos).
4. Install the base CNI plugins by placing the contents of `https://github.com/containernetworking/plugins/releases/download/v1.0.0/cni-plugins-linux-amd64-v1.0.0.tgz`into `/opt/cni/bin`
5. `systemctl start docker`
6. `systemctl start nomad`
7. Run literally any job (I've included my job file below but we've seen this happen with many jobs)
8. `systemctl restart docker`

#### Expected Result

The ip/port combo that the job binds should be `curl`-able. It is before docker is restarted.

#### Actual Result

If you curl the ip/port combo it will complain about having no route to host:

```
root@client-1:~# curl -v localhost:27846
* Trying 127.0.0.1:27846...
* Trying ::1:27846...
* connect to ::1 port 27846 failed: Connection refused
* connect to 127.0.0.1 port 27846 failed: No route to host
* Failed to connect to localhost port 27846 after 3061 ms: No route to host
* Closing connection 0
curl: (7) Failed to connect to localhost port 27846 after 3061 ms: No route to host
```

This makes sense as executing `ip addr` from within the container will now reveal the container has lost it's bridge network veth.

### Job file (if appropriate)

We've noticed this happen with every job but the job file I used for the reproduction is:
```
job "jess-test-job" {
type = "service"
datacenters = ["*"]
group "http" {
network {
mode = "bridge"
port "http" {
to = "80"
}
}
task "whoami" {
driver = "docker"
config {
image = "strm/helloworld-http"
ports = ["http"]
}
}
}
}
```

The toy instance I used for reproduction has a broken journal so sadly I have no logs from that to provide. If reproduction turns out to be an issue I'd be happy to send over some logs from one of our actual failing instances but I have a hunch this won't be that hard to reproduce.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the failure using the Ubuntu 22.04, Docker, Nomad, CNI plugin, and bridge-mode job steps in the report, then inspect how Nomad and Docker handle container networking after a daemon restart. Done means restarted containers retain their bridge interfaces and the job's advertised IP and port remain reachable.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
devops, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.