hashicorp / hashicorp/nomad

Nomad Failing To Create Job when Docker "userns-remap": "default"

Open
#8,459 17 comments 1 reaction 0 assignees View on GitHub
stage/accepted stage/needs-verification theme/driver/docker type/enhancement
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

If filing a bug please include the following:

### Nomad version
`Nomad v0.12.0 (8f7fbc8e7b5a4ed0d0209968faf41b238e6d5817)`

### Operating system and Environment details
```
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.6 LTS
Release: 16.04
Codename: xenial
```
### Issue
By default, we run our docker daemon with `userns-remap=default`. In this case, even the simplest Job file (e.g., from `nomad init -short`) is failing

Docker Daemon File
```json
{
"hosts": [ "unix:///var/run/docker.sock", "tcp://0.0.0.0:2376" ],
"labels": [ "is-our-remote-engine=true" ], "tls": true, "tlsverify": true,
"tlscacert": "/etc/docker/ca.pem",
"tlscert": "/etc/docker/cert.pem",
"tlskey": "/etc/docker/key.pem",
"data-root": "/data/docker-storage",
"storage-driver": "overlay2",
"log-driver": "json-file",
"log-opts": {
"max-size": "1m",
"max-file": "10"
},
"userns-remap": "default",
"bip": "192.168.2.1/24"
}
```

### Reproduction steps
* Turn userns=default on docker daeon
* Ensure you have valid subuid/subgid files
```bash
$ cat /etc/subgid
dockremap:100000:65536
$cat /etc/subuid
dockremap:100000:65536
```
* Deploy job from `nomad init -short`

### Job file (if appropriate)
```hcl
job "example" {
datacenters = ["dc1"]

group "cache" {
task "redis" {
driver = "docker"

config {
image = "redis:3.2"

port_map {
db = 6379
}
}

resources {
cpu = 500
memory = 256

network {
mbits = 10
port "db" {}
}
}
}
}
}
```

### Nomad Client logs (if appropriate)
Nomad Log Snippet
```

client.driver_mgr.docker: created container: driver=docker container_id=fec6cb88295e45fcaad69e99cf96c134ff4a7ff6336c002080d0d45b2e34e205
client.driver_mgr.docker: failed to start container: driver=docker container_id=fec6cb88295e45fcaad69e99cf96c134ff4a7ff6336c002080d0d45b2e34e205 ntainer_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\\"/data/nomad/alloc/tfs \\\"/data/docker-storage/100000.100000/overlay2/8905d8ffa72be2d9f0c97b4971e5b50b079b9a46a200bdfac6d255117f5f26fd/merged\\\" at \\\"/alloc\\\" caused 5e3cab1b0d/alloc: permission denied\\\"\"": unknown"
2020-07-17T12:14:27.679-0700 [ERROR] client.driver_mgr.docker: failed to start container: driver=docker 336c002080d0d45b2e34e205 error="API error (400): OCI runtime create failed: container_linux.go:349: starting container process caused _linux.go:58: mounting \\\"/data/nomad/alloc/f79d1ebb-31d7-673c-a80f-dc5e3cab1b0d/alloc\\\" to rootfs \\\"/data/docker-storage/100000.100000/bdfac6d255117f5f26fd/merged\\\" at \\\"/alloc\\\" caused \\\"stat /data/nomad/alloc/f79d1ebb-31d7-673c-a80f-dc5e3cab1b0d/alloc: permission denied\\\"\"":
2020-07-17T12:14:27.685-0700 [ERROR] client.alloc_runner.task_runner: running driver failed: alloc_id=f79d1ebb-31d7-673c-a80f-dc5e3cab1b0d task=redis 9e99cf96c134ff4a7ff6336c002080d0d45b2e34e205: API error (400): OCI runtime create failed: container_linux.go:349: starting container process caused _linux.go:58: mounting \\\"/data/nomad/alloc/f79d1ebb-31d7-673c-a80f-dc5e3cab1b0d/alloc\\\" to rootfs \\\"/data/docker-storage/100000.100000/bdfac6d255117f5f26fd/merged\\\" at \\\"/alloc\\\" caused \\\"stat /data/nomad/alloc/f79d1ebb-31d7-673c-a80f-dc5e3cab1b0d/alloc: permission denied\\\"\"":
2020-07-17T12:14:27.685-0700 [INFO] client.alloc_runner.task_runner: not restarting task: alloc_id=f79d1ebb-31d7-673c-a80f-dc5e3cab1b0d task=redis
client.alloc_runner.task_runner: running driver failed: alloc_id=f79d1ebb-31d7-673c-a80f-dc5e3cab1b0d task=redis error="Failed to start container 45b2e34e205: API error (400): OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init /alloc/f79d1ebb-31d7-673c-a80f-dc5e3cab1b0d/alloc\\\" to rootfs \\\"/data/docker-storage/100000.100000/bdfac6d255117f5f26fd/merged\\\" at \\\"/alloc\\\" caused \\\"stat /data/nomad/alloc/f79d1ebb-31d7-673c-a80f-dc5e3cab1b0d/alloc: permission denied\\\"\"":
client.alloc_runner.task_runner: not restarting task: alloc_id=f79d1ebb-31d7-673c-a80f-dc5e3cab1b0d task=redis reason="Error was unrecoverable"
2020-07-17T12:14:27.686-0700 [INFO] client.gc: marking allocation for GC: alloc_id=f79d1ebb-31d7-673c-a80f-dc5e3cab1b0d
client.gc: marking allocation for GC: alloc_id=f79d1ebb-31d7-673c-a80f-dc5e3cab1b0d
```

It looks like root own `/data/nomad/alloc/f79d1ebb-31d7-673c-a80f-dc5e3cab1b0d/alloc` but `/data/docker-storage/100000.100000/bdfac6d255117f5f26fd/merged\\\` is of course user namespaced

In order to make things simple - I am running the client Nomad node as root (although would prefer to add it to docker group later on).

Any ideas?

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the failure with Docker userns-remap=default, the supplied subuid/subgid entries, and the job from `nomad init -short`. Start at the Docker driver path that creates and starts the container, focusing on the allocation mount shown in the logs. Done means the example job starts successfully under this configuration, with a regression test if the relevant test entry point is found.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.