Nomad Failing To Create Job when Docker "userns-remap": "default"
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
If filing a bug please include the following:
### Nomad version
`Nomad v0.12.0 (8f7fbc8e7b5a4ed0d0209968faf41b238e6d5817)`
### Operating system and Environment details
```
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.6 LTS
Release: 16.04
Codename: xenial
```
### Issue
By default, we run our docker daemon with `userns-remap=default`. In this case, even the simplest Job file (e.g., from `nomad init -short`) is failing
Docker Daemon File
```json
{
"hosts": [ "unix:///var/run/docker.sock", "tcp://0.0.0.0:2376" ],
"labels": [ "is-our-remote-engine=true" ], "tls": true, "tlsverify": true,
"tlscacert": "/etc/docker/ca.pem",
"tlscert": "/etc/docker/cert.pem",
"tlskey": "/etc/docker/key.pem",
"data-root": "/data/docker-storage",
"storage-driver": "overlay2",
"log-driver": "json-file",
"log-opts": {
"max-size": "1m",
"max-file": "10"
},
"userns-remap": "default",
"bip": "192.168.2.1/24"
}
```
### Reproduction steps
* Turn userns=default on docker daeon
* Ensure you have valid subuid/subgid files
```bash
$ cat /etc/subgid
dockremap:100000:65536
$cat /etc/subuid
dockremap:100000:65536
```
* Deploy job from `nomad init -short`
### Job file (if appropriate)
```hcl
job "example" {
datacenters = ["dc1"]
group "cache" {
task "redis" {
driver = "docker"
config {
image = "redis:3.2"
port_map {
db = 6379
}
}
resources {
cpu = 500
memory = 256
network {
mbits = 10
port "db" {}
}
}
}
}
}
```
### Nomad Client logs (if appropriate)
Nomad Log Snippet
```
client.driver_mgr.docker: created container: driver=docker container_id=fec6cb88295e45fcaad69e99cf96c134ff4a7ff6336c002080d0d45b2e34e205
client.driver_mgr.docker: failed to start container: driver=docker container_id=fec6cb88295e45fcaad69e99cf96c134ff4a7ff6336c002080d0d45b2e34e205 ntainer_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\\"/data/nomad/alloc/tfs \\\"/data/docker-storage/100000.100000/overlay2/8905d8ffa72be2d9f0c97b4971e5b50b079b9a46a200bdfac6d255117f5f26fd/merged\\\" at \\\"/alloc\\\" caused 5e3cab1b0d/alloc: permission denied\\\"\"": unknown"
2020-07-17T12:14:27.679-0700 [ERROR] client.driver_mgr.docker: failed to start container: driver=docker 336c002080d0d45b2e34e205 error="API error (400): OCI runtime create failed: container_linux.go:349: starting container process caused _linux.go:58: mounting \\\"/data/nomad/alloc/f79d1ebb-31d7-673c-a80f-dc5e3cab1b0d/alloc\\\" to rootfs \\\"/data/docker-storage/100000.100000/bdfac6d255117f5f26fd/merged\\\" at \\\"/alloc\\\" caused \\\"stat /data/nomad/alloc/f79d1ebb-31d7-673c-a80f-dc5e3cab1b0d/alloc: permission denied\\\"\"":
2020-07-17T12:14:27.685-0700 [ERROR] client.alloc_runner.task_runner: running driver failed: alloc_id=f79d1ebb-31d7-673c-a80f-dc5e3cab1b0d task=redis 9e99cf96c134ff4a7ff6336c002080d0d45b2e34e205: API error (400): OCI runtime create failed: container_linux.go:349: starting container process caused _linux.go:58: mounting \\\"/data/nomad/alloc/f79d1ebb-31d7-673c-a80f-dc5e3cab1b0d/alloc\\\" to rootfs \\\"/data/docker-storage/100000.100000/bdfac6d255117f5f26fd/merged\\\" at \\\"/alloc\\\" caused \\\"stat /data/nomad/alloc/f79d1ebb-31d7-673c-a80f-dc5e3cab1b0d/alloc: permission denied\\\"\"":
2020-07-17T12:14:27.685-0700 [INFO] client.alloc_runner.task_runner: not restarting task: alloc_id=f79d1ebb-31d7-673c-a80f-dc5e3cab1b0d task=redis
client.alloc_runner.task_runner: running driver failed: alloc_id=f79d1ebb-31d7-673c-a80f-dc5e3cab1b0d task=redis error="Failed to start container 45b2e34e205: API error (400): OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init /alloc/f79d1ebb-31d7-673c-a80f-dc5e3cab1b0d/alloc\\\" to rootfs \\\"/data/docker-storage/100000.100000/bdfac6d255117f5f26fd/merged\\\" at \\\"/alloc\\\" caused \\\"stat /data/nomad/alloc/f79d1ebb-31d7-673c-a80f-dc5e3cab1b0d/alloc: permission denied\\\"\"":
client.alloc_runner.task_runner: not restarting task: alloc_id=f79d1ebb-31d7-673c-a80f-dc5e3cab1b0d task=redis reason="Error was unrecoverable"
2020-07-17T12:14:27.686-0700 [INFO] client.gc: marking allocation for GC: alloc_id=f79d1ebb-31d7-673c-a80f-dc5e3cab1b0d
client.gc: marking allocation for GC: alloc_id=f79d1ebb-31d7-673c-a80f-dc5e3cab1b0d
```
It looks like root own `/data/nomad/alloc/f79d1ebb-31d7-673c-a80f-dc5e3cab1b0d/alloc` but `/data/docker-storage/100000.100000/bdfac6d255117f5f26fd/merged\\\` is of course user namespaced
In order to make things simple - I am running the client Nomad node as root (although would prefer to add it to docker group later on).
Any ideas?
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the failure with Docker userns-remap=default, the supplied subuid/subgid entries, and the job from `nomad init -short`. Start at the Docker driver path that creates and starts the container, focusing on the allocation mount shown in the logs. Done means the example job starts successfully under this configuration, with a regression test if the relevant test entry point is found.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100