Exec driver fails to run command with 'permission denied' error when umask restricts world execute (077)
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
Nomad v0.5.6
### Operating system and Environment details
Ubuntu 16.04
### Issue
As part of our OS hardening, we set the umask to 077 (rather than default of 022). With this umask value, the alloc directory (and possibly others further down the tree) has the permissions value 700 instead of the [intended 755](https://github.com/hashicorp/nomad/blob/master/client/allocdir/alloc_dir.go#L256). When the default 'nobody' (or any non-root) user tries to execute the exec driver command, the directory is not accessible and so the job fails with a 'permission denied' error.
### Job file
Saved as example.nomad
```
job "example" {
datacenters = ["dc1"]
group "example" {
task "example" {
driver = "exec"
config {
command = "/bin/sleep"
args = ["10"]
}
}
}
}
```
### Reproduction steps
Within the Vagrant environment:
```
ubuntu@nomad:~$ sudo su
root@nomad:/home/ubuntu# umask
0022
root@nomad:/home/ubuntu# umask 0077
root@nomad:/home/ubuntu# umask
0077
root@nomad:/home/ubuntu# nomad agent -dev &
...
2017/06/21 08:48:26.380046 [INFO] client: using state directory /tmp/NomadClient406600935
2017/06/21 08:48:26.380084 [INFO] client: using alloc directory /tmp/NomadClient992233242
...
root@nomad:/home/ubuntu# nomad run example.nomad
...
2017/06/21 08:48:50 [DEBUG] plugin: starting plugin: /usr/bin/nomad []string{"/usr/bin/nomad", "executor", "{\"LogFile\":\"/tmp/NomadClient992233242/b4015e2e-f495-dce7-be8e-a59bfde20d85/example/executor.out\",\"LogLevel\":\"DEBUG\"}"}
2017/06/21 08:48:50 [DEBUG] plugin: waiting for RPC address for: /usr/bin/nomad
2017/06/21 08:48:50 [DEBUG] plugin: nomad: 2017/06/21 08:48:50 [DEBUG] plugin: plugin address: unix /tmp/plugin889551382
2017/06/21 08:48:50 [DEBUG] plugin: /usr/bin/nomad: plugin process exited
2017/06/21 08:48:50.780090 [WARN] client: failed to start task "example" for alloc "b4015e2e-f495-dce7-be8e-a59bfde20d85": failed to start command path="bin/sleep" --- args=["bin/sleep" "10"]: fork/exec bin/sleep: permission denied
```
Note the lack of the executable bit for the non-root (nobody) user on the alloc directory
```
root@nomad:/home/ubuntu# ls -ltrah /tmp/NomadClient992233242
total 12K
drwx------ 4 root root 4.0K Jun 21 08:48 b4015e2e-f495-dce7-be8e-a59bfde20d85
drwxr-xr-x 3 root root 4.0K Jun 21 08:48 .
drwxrwxrwt 9 root root 4.0K Jun 21 08:48 ..
```
Changing umask back to 0022, restarting Nomad and resubmitting the job is successful:
```
root@nomad:/home/ubuntu# umask
0077
root@nomad:/home/ubuntu# umask 0022
root@nomad:/home/ubuntu# umask
0022
root@nomad:/home/ubuntu# nomad agent -dev &
...
2017/06/21 08:51:28.082829 [INFO] client: using state directory /tmp/NomadClient568517847
2017/06/21 08:51:28.082908 [INFO] client: using alloc directory /tmp/NomadClient038854218
...
root@nomad:/home/ubuntu# nomad run example.nomad
...
2017/06/21 08:52:09.685175 [INFO] client: task "example" for alloc "38d42234-38d9-cfeb-a331-fd8b736c7918" completed successfully
```
New alloc directory has the executable bit for the non-root (nobody) user
```
root@nomad:/home/ubuntu# ls -ltrah /tmp/NomadClient038854218
total 12K
drwxr-xr-x 3 root root 4.0K Jun 21 08:51 .
drwxr-xr-x 4 root root 4.0K Jun 21 08:51 38d42234-38d9-cfeb-a331-fd8b736c7918
drwxrwxrwt 11 root root 4.0K Jun 21 08:52 ..
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in client/allocdir/alloc_dir.go at the linked directory-permission logic and reproduce the issue with umask 077 from the report. Check that allocation directories remain accessible to the non-root exec user and that the example exec job runs successfully; verify the behavior under both restrictive and default umasks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, operating-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100