hashicorp / hashicorp/nomad

inconsistent behavior for /etc/resolv.conf with stub resolvers

Open
#11,033 3 comments 2 reactions 0 assignees View on GitHub
stage/accepted theme/driver theme/networking type/enhancement
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

While chatting with @mattrobenolt about exposing Consul DNS to a Nomad task, we ran into something unexpected where the behavior I described in https://github.com/hashicorp/nomad/issues/8343#issuecomment-863325421 doesn't hold in the case where the client is not using `systemd-resolved`. It turns out that `dockerd` (or more properly `libnetwork`) special-cases how it creates a `/etc/resolv.conf` file for containers when it thinks `systemd-resolved` is in use. See [`moby/libnetwork/resolvconf/resolvconf.go#L18-L21`](https://github.com/moby/libnetwork/blob/64b7a4574d1426139437d20e81c0b6d391130ec8/resolvconf/resolvconf.go#L18-L21)

* with `systemd-resolved`, `docker` tasks get the host's (non-stub) `resolv.conf`.
* with `systemd-resolved`, `exec` tasks get the stub `resolv.conf` pointing to `127.0.0.53`.
* without `systemd-resolved`, both `docker` and `exec` tasks get the host's `resolv.conf`.
* with a `network.dns` block, both `docker` and `exec` tasks get the Nomad-managed `resolv.conf` generated by [`GenerateDNSMount`](https://github.com/hashicorp/nomad/blob/8e33fc550398e865ea253b18e723110267b782ab/drivers/shared/resolvconf/mount.go#L13)

While the inconsistent behavior is the "fault" of the task driver engine, it makes configuring Consul DNS for tasks in a sensible way challenging and awfully host-specific.

Some proposals:
* Can Nomad _always_ manage the `resolv.conf`? This would break backwards compatibility but would make it possible to point tasks to a Nomad-controlled IP; this could be the host's IP or a Consul IP for a stub resolver, etc.
* Can `exec` task drivers do the same thing that `docker` does with the stub resolver? (This is kind of gross and is annoyingly undocumented in Docker, too, as far as we can tell).
* In lieu of that, a couple of "blessed" configurations for Consul DNS would be super nice to have documented.

---

### Reproduction for the stub resolver behavior.

Using the Vagrant machine found at the root of this repo and running the following jobspec that has both a `docker` and `exec` task sharing a network namespace:

Docker jobspec

```hcl
job "example" {
datacenters = ["dc1"]

group "web" {
network {
mode = "bridge"
port "web1" {
to = 8001
}
port "web2" {
to = 8002
}
}

task "web1" {
driver = "docker"

config {
image = "busybox:1"
command = "httpd"
args = ["-f", "-h", "/tmp", "-p", "8001"]
ports = ["web1"]
}

resources {
cpu = 256
memory = 128
}
}

task "web2" {
driver = "exec"

config {
command = "busybox"
args = ["httpd", "-f", "-h", "/tmp", "-p", "8002"]
}

resources {
cpu = 256
memory = 128
}
}

}
}
```

The `docker` driver gets the "real" resolver used by systemd-resolved and found at `/run/systemd/resolve/resolv.conf`:

```
$ nomad alloc exec -task web1 d2f cat /etc/resolv.conf
...

nameserver 10.0.2.3
search fios-router.home
```

But the `exec` driver gets the stub resolver from the host's `/etc/resolv.conf`

```
$ nomad alloc exec -task web2 d2f cat /etc/resolv.conf
...

nameserver 127.0.0.53
options edns0
search fios-router.home
```

Now replace systemd-resolved's stub resolver with unbound:

```
$ sudo apt-get install -y unbound
$ echo 'DNSStubListener=no' | sudo tee /etc/systemd/resolved.conf
```

Restart the VM and make sure the stub listener isn't what's linked to `/etc/resolv.conf` anymore:

```
sudo systemctl stop systemd-resolved
sudo rm /etc/resolv.conf
echo 'nameserver 8.8.8.8' | sudo tee /etc/resolv.conf
sudo systemctl start systemd-resolved
```

Now both of them get the `/etc/resolv.conf` file from the host:

```
$ nomad alloc exec -task web1 208 cat /etc/resolv.conf
nameserver 8.8.8.8

$ nomad alloc exec -task web2 208 cat /etc/resolv.conf
nameserver 8.8.8.8
```

restore your Vagrant VM back to the previous state

```
sudo rm /etc/resolv.conf
sudo nano /etc/systemd/resolved.conf # remove the DNSStubListener=no line
sudo apt-get remove -y unbound
sudo ln -fs /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf
```

Then reboot the VM.

Contributor guide

No contributing guide indexed for this repository

Research direction

Begin with drivers/shared/resolvconf/mount.go and the Docker/libnetwork resolvconf behavior, then reproduce the difference using the root Vagrant machine and the jobspec in this issue. Done requires an agreed behavior for docker and exec tasks, or an agreed documentation-only outcome; the issue does not select one.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go, linux
Domain
infrastructure, networking
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.