Consul ACL token is created as root:root 0640 and is unreadable by non-root Docker task user
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
Nomad 2.0.4 on servers and clients.
Consul 1.22.7
This was first observed after upgrading from Nomad 1.11.3 to 2.0.4. We have not yet reproduced this specific ownership behavior on 1.11.3, so it is not confirmed as a regression.
### Operating system and Environment details
- RHEL 8 compatible host
- Kernel: `4.18.0-553.139.1.el8_10.x86_64`
- Docker task driver
- Consul 1.22.7
- Consul ACLs enabled
- Nomad workload identities used for Consul authentication
-
### Nomad server configuration
All Nomad servers use the following Consul workload identity configuration:
```hcl
consul {
address = "localhost:8501"
ssl = true
ca_file = "/opt/consul/tls/ca.crt"
cert_file = "/opt/consul/tls/agent.crt"
key_file = "/opt/consul/tls/agent.key"
verify_ssl = true
grpc_address = "localhost:8503"
grpc_ca_file = "/opt/consul/tls/ca.crt"
service_identity {
aud = ["inf293.consul"]
ttl = "1h"
}
task_identity {
aud = ["inf293.consul"]
ttl = "1h"
env = false
file = true
}
}
```
The Consul JWT auth method and binding rules are working. Nomad successfully exchanges the workload identity for a Consul ACL token.
#### Expected Result
The Consul ACL token written to `secrets/consul_token` should be readable by the configured task user.
Since the task explicitly runs as nobody, we expected either:
`nobody: 0600 consul_token`
or another ownership/mode combination that permits UID 65534 to read the token without granting additional groups or running the task as root.
Nomad documents `secrets/consul_token` as being made available to the task. A non-root task should therefore be able to consume it through an application's credentials_file option.
#### Actual Result
Nomad creates the files as follows:
```
-rw-r--r-- 1 root root 95 vault_token
-rw------- 1 nobody root 947 nomad_vault_default.jwt
-rw------- 1 nobody root 920 nomad_consul_default.jwt
-rw-r----- 1 root root 36 consul_token
srw------- 1 nobody root 0 api.sock
```
The task runs as:
`uid=65534(nobody) gid=65534(nobody) groups=65534(nobody)`
The workload identity JWT files correctly use the configured task user as their owner, but the exchanged Consul ACL token does not.
Applications that consume the generated token through a credentials file fail with:
```
unable to read authorization credentials:
unable to read file /secrets/consul_token:
open /secrets/consul_token: permission denied
```
### Current workaround
For Docker tasks we can add GID 0 as a supplementary group:
```
config {
image = "prom/prometheus:v2.55.1"
group_add = ["0"]
}
```
The process remains non-root:
`uid=65534(nobody) gid=65534(nobody) groups=0(root),65534(nobody)`
and can then read root:root 0640 files.
However, this broadens the task's access to every root-group-readable bind mount and secret. We do not consider adding the root group to arbitrary non-root workloads an appropriate permanent solution.
Running the application as root would be an even less desirable workaround.
### Job file (if appropriate)
A minimal reproduction should be possible with:
```
job "consul-token-permissions" {
datacenters = ["dc1"]
type = "service"
group "test" {
task "test" {
driver = "docker"
user = "nobody"
consul {}
config {
image = "busybox:1.36"
command = "/bin/sh"
args = [
"-ec",
<<-EOF
id
ls -ln /secrets
stat -c '%U %G %a %n' /secrets/consul_token
test -r /secrets/consul_token
sleep 3600
EOF
]
}
}
}
}
```
thanks,
mac
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the minimal Docker task reproduction in the issue and inspect the ownership and permissions of /secrets/consul_token alongside the workload identity JWT files. The fix is done when a task running as nobody can read the exchanged Consul token without adding the root group or running as root, while the existing JWT behavior remains correct.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go, linux
- Domain
- authentication, backend, security
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 62/100