hashicorp / hashicorp/nomad

Consul ACL token is created as root:root 0640 and is unreadable by non-root Docker task user

Open
#28,347 3 comments 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
Nomad 2.0.4 on servers and clients.
Consul 1.22.7

This was first observed after upgrading from Nomad 1.11.3 to 2.0.4. We have not yet reproduced this specific ownership behavior on 1.11.3, so it is not confirmed as a regression.

### Operating system and Environment details

- RHEL 8 compatible host
- Kernel: `4.18.0-553.139.1.el8_10.x86_64`
- Docker task driver
- Consul 1.22.7
- Consul ACLs enabled
- Nomad workload identities used for Consul authentication
-
### Nomad server configuration

All Nomad servers use the following Consul workload identity configuration:

```hcl
consul {
address = "localhost:8501"
ssl = true
ca_file = "/opt/consul/tls/ca.crt"
cert_file = "/opt/consul/tls/agent.crt"
key_file = "/opt/consul/tls/agent.key"
verify_ssl = true
grpc_address = "localhost:8503"
grpc_ca_file = "/opt/consul/tls/ca.crt"

service_identity {
aud = ["inf293.consul"]
ttl = "1h"
}

task_identity {
aud = ["inf293.consul"]
ttl = "1h"
env = false
file = true
}
}
```

The Consul JWT auth method and binding rules are working. Nomad successfully exchanges the workload identity for a Consul ACL token.

#### Expected Result

The Consul ACL token written to `secrets/consul_token` should be readable by the configured task user.
Since the task explicitly runs as nobody, we expected either:
`nobody: 0600 consul_token`
or another ownership/mode combination that permits UID 65534 to read the token without granting additional groups or running the task as root.
Nomad documents `secrets/consul_token` as being made available to the task. A non-root task should therefore be able to consume it through an application's credentials_file option.

#### Actual Result
Nomad creates the files as follows:
```
-rw-r--r-- 1 root root 95 vault_token
-rw------- 1 nobody root 947 nomad_vault_default.jwt
-rw------- 1 nobody root 920 nomad_consul_default.jwt
-rw-r----- 1 root root 36 consul_token
srw------- 1 nobody root 0 api.sock
```
The task runs as:
`uid=65534(nobody) gid=65534(nobody) groups=65534(nobody)`

The workload identity JWT files correctly use the configured task user as their owner, but the exchanged Consul ACL token does not.
Applications that consume the generated token through a credentials file fail with:
```
unable to read authorization credentials:
unable to read file /secrets/consul_token:
open /secrets/consul_token: permission denied
```

### Current workaround
For Docker tasks we can add GID 0 as a supplementary group:
```
config {
image = "prom/prometheus:v2.55.1"
group_add = ["0"]
}
```
The process remains non-root:
`uid=65534(nobody) gid=65534(nobody) groups=0(root),65534(nobody)`
and can then read root:root 0640 files.

However, this broadens the task's access to every root-group-readable bind mount and secret. We do not consider adding the root group to arbitrary non-root workloads an appropriate permanent solution.
Running the application as root would be an even less desirable workaround.

### Job file (if appropriate)
A minimal reproduction should be possible with:
```
job "consul-token-permissions" {
datacenters = ["dc1"]
type = "service"

group "test" {
task "test" {
driver = "docker"
user = "nobody"

consul {}

config {
image = "busybox:1.36"
command = "/bin/sh"
args = [
"-ec",
<<-EOF
id
ls -ln /secrets
stat -c '%U %G %a %n' /secrets/consul_token
test -r /secrets/consul_token
sleep 3600
EOF
]
}
}
}
}
```

thanks,
mac

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the minimal Docker task reproduction in the issue and inspect the ownership and permissions of /secrets/consul_token alongside the workload identity JWT files. The fix is done when a task running as nobody can read the exchanged Consul token without adding the root group or running as root, while the existing JWT behavior remains correct.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go, linux
Domain
authentication, backend, security
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
62/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.