hashicorp / hashicorp/nomad

Nomad-owned sockets not accessible under SELinux by default (Task API, CSI, Connect)

Open
#26,852 1 comment 0 reactions 0 assignees View on GitHub
theme/api theme/driver/docker theme/security theme/storage type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

When running Nomad under SELinux in enforcing mode, the Task API socket is not available, even if the appropriate labels are set in the plugin configuration. It appears this is also the case for the CSI plugin socket.

In the issue below I've got some configuration options for enabling it via SELinux policy but that opens a little bit of a hole. I don't think it's something we can solve in Nomad itself without risking interfering with SELinux policies that cluster admins want. (Unless we just set the socket to be `container_t`?) So maybe this needs to be something we document in the install/deploy guide? I think we'd definitely welcome the opinions of folks in the community who have specific interest here.

example jobspec

```hcl
job "curl" {

group "group" {

task "task" {

driver = "docker"

config {
image = "curlimages/curl:latest"
command = "tail"
args = ["-f"]
}

# this should you hit the Task API via alloc exec with:
# curl --unix-socket /secrets/api.sock \
# -H "X-Nomad-Token: $NOMAD_TOKEN" \
# "http://local/v1/..."
identity {
env = true
file = true
}

resources {
cpu = 128
memory = 256
}

}
}
}
```

We have the `volumes.selinuxlabel` configuration for the Docker plugin:

> [selinuxlabel](https://developer.hashicorp.com/nomad/docs/deploy/task-driver/docker#selinuxlabel) - Allows the operator to set a SELinux label to the allocation and task local bind-mounts to containers. If used with docker.volumes.enabled set to false, the labels will still be applied to the standard binds in the container.

Client configuration for the plugin:

```hcl
plugin "docker" {
config {
volumes {
enabled = true
selinuxlabel = "z"
}
}
}
```

If you `alloc exec` into this allocation and curl the Task API socket, you get a permission denied error in the container:

```
$ nomad alloc exec 0c4d805f /bin/sh

~ $ curl -v --unix-socket /secrets/api.sock -H "X-Nomad-Token: $NOMAD_TOKEN" "http://local/v1/vars"
* Trying /secrets/api.sock:0...
* Immediate connect fail for /secrets/api.sock: Permission denied
* Failed to connect to local port 80 after 0 ms: Could not connect to server
* closing connection #0
curl: (7) Failed to connect to local port 80 after 0 ms: Could not connect to server
```

And the following in your audit logs:

> type=AVC msg=audit(1759168401.396:2106): avc: denied { connectto } for pid=74015 comm="curl" path="/var/nomad/dev/data/alloc/0c4d805f-3edc-47e4-1e4f-a7d5dc918a6b/task/secrets/api.sock" scontext=system_u:system_r:container_t:s0:c39,c317 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=unix_stream_socket permissive=0

You can write to other locations in the secrets dir and the mount's `Mode` is configured correctly in the Docker container:

```
$ docker inspect 5b1ddb430132 | jq '.[0].Mounts[2]'
{
"Type": "bind",
"Source": "/var/nomad/dev/data/alloc/0c4d805f-3edc-47e4-1e4f-a7d5dc918a6b/task/secrets",
"Destination": "/secrets",
"Mode": "z",
"RW": true,
"Propagation": "rprivate"
}
```

You can bypass this problem by setting `task.config.security_opt = ["label=disable"]` but that's just bypassing SELinux protection entirely.

Here's what the resulting labels of objects in the secrets directory are:

```
# ls -lZ task/secrets/
total 8
srw-rw-rw-. 1 root root system_u:object_r:container_file_t:s0 0 Sep 29 13:43 api.sock
-rw-r--r--. 1 101 102 system_u:object_r:container_file_t:s0 8 Sep 29 13:45 foo.txt
-rw-r--r--. 1 root root system_u:object_r:container_file_t:s0 816 Sep 29 13:43 nomad_token
```

By my understanding of SELinux config, the `container_t` process that is the container should be able to write to the sockets that are `container_t`:

```
$ sesearch --allow -s container_t | grep unix_stream_socket
...
allow container_t container_t:unix_stream_socket { accept append bind connect connectto create getattr getopt ioctl listen lock map read sendto setattr setopt shutdown write };
...
```

We can use `audit2allow` to generate a SELinux policy file that allows access to the task API socket as follows. First, in a root shell verify that we've got the correct item in the audit log:

```
root:/tmp# tail -1 /var/log/audit/audit.log | grep container_t
type=AVC msg=audit(1759169470.561:2169): avc: denied { connectto } for pid=78079 comm="curl" path="/var/nomad/dev/data/alloc/0c4d805f-3edc-47e4-1e4f-a7d5dc918a6b/task/secrets/api.sock" scontext=system_u:system_r:container_t:s0:c39,c317 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=unix_stream_socket permissive=0
```

Generate a type enforcement policy file (which by my understanding is really only for human review). This shows the the socket was actually of `unconfined_t`, which doesn't make a lot of sense to me given the output of `ls -lZ` above:

```
root:/tmp# tail -1 /var/log/audit/audit.log | grep container_t | audit2allow -m nomad_task_api > ./nomad_task_api.te

root:/tmp# cat ./nomad_task_api.te

module nomad_task_api 1.0;

require {
type container_t;
type unconfined_t;
class unix_stream_socket connectto;
}
```

If we generate the policy and install it:

```
root:/tmp# tail -1 /var/log/audit/audit.log | grep container_t | audit2allow -M nomad_task_api
******************** IMPORTANT ***********************
To make this policy package active, execute:

semodule -i nomad_task_api.pp

root:/tmp# semodule -i ./nomad_task_api.pp
```

At this point our Task API socket works:

```
~ $ curl -v --unix-socket /secrets/api.sock -H "X-Nomad-Token: $NOMAD_TOKEN" "http://local/v1/vars"
* Trying /secrets/api.sock:0...
* Established connection to local (/secrets/api.sock port 0) from port 0
* using HTTP/1.x
> GET /v1/vars HTTP/1.1
> Host: local
> User-Agent: curl/8.16.0
> Accept: */*
> X-Nomad-Token: eyJhbGc[REDACTED]
>
* Request completely sent off
< HTTP/1.1 200 OK
< Content-Type: application/json
< Vary: Origin
< X-Nomad-Index: 1
< X-Nomad-Knownleader: true
< X-Nomad-Lastcontact: 0
< Date: Mon, 29 Sep 2025 18:12:37 GMT
< Content-Length: 2
<
* Connection #0 to host local:80 left intact
[]~ $ ^C
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the Docker task-driver SELinux documentation linked in the issue and review the reported Task API socket behavior under enforcing mode. Compare the implications for the Task API, CSI, and Connect sockets, including the audit2allow and label-disable workarounds. Done means the supported configuration or policy boundary is decided and documented safely.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, linux
Domain
devops, infrastructure, security
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.