Host address unreachable from exec driver if docker is present on host
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
Output from `nomad version`
```
Nomad v1.3.1 (2b054e38e91af964d1235faa98c286ca3f527e56)
```
### Operating system and Environment details
```bash
No LSB modules are available.
Distributor ID: Pop
Description: Pop!_OS 22.04 LTS
Release: 22.04
Codename: jammy
```
### Issue
1) On a fresh Nomad client VM, I deploy an exec job which is similar to:
```hcl
job "http" {
datacenters = ["dc1"]
type = "service"
group "app" {
count = 1
network {
mode = "bridge"
port "python-http" {
to = "8888"
}
}
task "server" {
driver = "exec"
config {
command = "/usr/bin/python3"
args = ["-m", "http.server", "8888"]
}
}
}
}
```
2) The job gets deployed and I can see the `Host Address` inside Allocation:

3) I `exec` inside the alloc, and try to reach this address (`192.168.29.76:31958`):
```bash
$ nomad alloc exec -i -t -task server ff724b46 /bin/bash
nobody@pop-os:/$ curl 192.168.29.76:31958
Directory listing for /
Directory listing for /
nobody@pop-os:/$
```
4) I install docker on this host.
Now, since docker mangles `iptables` on the host, here's a snapshot of all the rules existing on this host:
```bash
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain NOMAD-ADMIN (0 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * nomad 0.0.0.0/0 172.26.64.0/20
```
After I [install docker](https://docs.docker.com/engine/install/ubuntu/#install-using-the-convenience-script), the above `curl` command stops working:
```bash
nobody@pop-os:/$ curl -m 5 192.168.29.76:24858
curl: (28) Connection timed out after 5004 milliseconds
nobody@pop-os:/$
```
The `iptables` rules list after `docker` is installed:
```bash
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 DOCKER-USER all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 DOCKER-ISOLATION-STAGE-1 all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
0 0 DOCKER all -- * docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- docker0 docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- * br-886858651ca3 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
0 0 DOCKER all -- * br-886858651ca3 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- br-886858651ca3 !br-886858651ca3 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- br-886858651ca3 br-886858651ca3 0.0.0.0/0 0.0.0.0/0
12 1747 CNI-FORWARD all -- * * 0.0.0.0/0 0.0.0.0/0 /* CNI firewall plugin rules */
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain CNI-FORWARD (1 references)
pkts bytes target prot opt in out source destination
12 1747 NOMAD-ADMIN all -- * * 0.0.0.0/0 0.0.0.0/0 /* CNI firewall plugin admin overrides */
0 0 ACCEPT all -- * * 0.0.0.0/0 172.26.64.87 ctstate RELATED,ESTABLISHED
0 0 ACCEPT all -- * * 172.26.64.87 0.0.0.0/0
Chain DOCKER (2 references)
pkts bytes target prot opt in out source destination
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
pkts bytes target prot opt in out source destination
0 0 DOCKER-ISOLATION-STAGE-2 all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
0 0 DOCKER-ISOLATION-STAGE-2 all -- br-886858651ca3 !br-886858651ca3 0.0.0.0/0 0.0.0.0/0
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-ISOLATION-STAGE-2 (2 references)
pkts bytes target prot opt in out source destination
0 0 DROP all -- * docker0 0.0.0.0/0 0.0.0.0/0
0 0 DROP all -- * br-886858651ca3 0.0.0.0/0 0.0.0.0/0
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-USER (1 references)
pkts bytes target prot opt in out source destination
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain NOMAD-ADMIN (1 references)
pkts bytes target prot opt in out source destination
12 1747 ACCEPT all -- * nomad 0.0.0.0/0 172.26.64.0/20
```
### Reproduction steps
Detailed steps are above already.
Here's a TL;DR:
- Run a fresh VM with no docker
- Run a task with exec driver and a port mapped to the host address
- Curl the address and see it works
- Install docker
- The above address is unreachable now.
## More context
I wonder if `docker` is putting some kind of iptables rule on the `host` network interface which makes it unreachable from the `nomad` network interface ? Which is why as soon as docker is installed on the host, the address is unreachable.
IP routes on the host:
```
default via 192.168.29.1 dev wlp0s20f3 proto dhcp metric 600
169.254.0.0/16 dev wlp0s20f3 scope link metric 1000
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
172.18.0.0/16 dev br-886858651ca3 proto kernel scope link src 172.18.0.1 linkdown
172.26.64.0/20 dev nomad proto kernel scope link src 172.26.64.1
192.168.29.0/24 dev wlp0s20f3 proto kernel scope link src 192.168.29.76 metric 600
```
IP routes on the alloc:
```
nobody@pop-os:/$ ip route
default via 172.26.64.1 dev eth0
172.26.64.0/20 dev eth0 proto kernel scope link src 172.26.64.87
nobody@pop-os:/$
```
(This I believe is the default [subnet](https://www.nomadproject.io/docs/configuration/client#bridge_network_subnet) that nomad uses).
**Question**:
What I want to achieve is to be able to reach the application from inside the `alloc exec` for quick debugging/tests. What is the best way to achieve that/which address/interface should I be using in that case? I've tried the `lo`/`0.0.0.0`/`nomad` but none seem to work. This is unlike docker driver where the application binds to 127.0.0.1 in the container itself so it's reachable, so how exactly would this work in `exec`?
Thanks!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the exec driver's bridge networking and the CNI-FORWARD/NOMAD-ADMIN iptables path, reproducing the curl failure before and after Docker installation. Done means identifying the supported address or interface, or a concrete Nomad change or documentation path that lets an exec allocation reach the host-mapped service.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, linux
- Domain
- networking, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100