hashicorp / hashicorp/nomad

Nomad connect functionality not working with SELinux enabled

Open
#7,290 11 comments 0 reactions 0 assignees View on GitHub
theme/consul/connect theme/docs type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
Nomad v0.10.4 (f750636ca68e17dcd2445c1ab9c5a34f9ac69345)

### Operating system and Environment details
Fedora 31, with 18.09.8

### Issue
The envoy health check in consul stays red and `var/log/audit/audit.log` contains denials:
```
type=AVC msg=audit(1583672022.178:2020): avc: denied { write } for pid=70868 comm="envoy" name="consul_grpc.sock" dev="tmpfs" ino=676989 scontext=system_u:system_r:container_t:s0:c121,c146 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=sock_file permissive=0
```

After `sentenforce 0` the health check turns green.

### Reproduction steps

Run `nomad agent -dev` and `consul agent -dev` and deploy the job file from below

### Job file (if appropriate)

```
job "example" {
datacenters = ["dc1"]
type = "service"
update { max_parallel = 1 }
group "http1" {
network {
mode = "bridge"
port "http" { to=80 }
}
service {
port = "http"
name = "http1"

connect {
sidecar_service {}
}
}
task "http1" {
driver = "docker"
config { image = "nginx" }
}
}
group "http2" {
network {
mode = "bridge"
port "http" { to=80 }
}
service {
port = "http"
name = "http2"

connect {
sidecar_service {
proxy {
upstreams {
destination_name = "http1"
local_bind_port = 8080
}
}
}
}

}
task "http2" {
driver = "docker"
config { image = "nginx" }
}
}
}
```

Consul logs have:
```
2020-03-08T14:02:56.962+0100 [WARN] agent: Check socket connection failed: check=service:_nomad-task-6d05e4c5-b5d8-2941-c6a4-dc9bb1e675c6-group-http2-http2-http-sidecar-proxy:1 error="dial tcp 127.0.0.1:30124: connect: connection refused"
2020-03-08T14:02:56.963+0100 [WARN] agent: Check is now critical: check=service:_nomad-task-6d05e4c5-b5d8-2941-c6a4-dc9bb1e675c6-group-http2-http2-http-sidecar-proxy:1
2020-03-08T14:03:00.781+0100 [WARN] agent: Check socket connection failed: check=service:_nomad-task-a6ae689e-6b3c-206d-5b58-0562248a595c-group-http1-http1-http-sidecar-proxy:1 error="dial tcp 127.0.0.1:20423: connect: connection refused"
2020-03-08T14:03:00.781+0100 [WARN] agent: Check is now critical: check=service:_nomad-task-a6ae689e-6b3c-206d-5b58-0562248a595c-group-http1-http1-http-sidecar-proxy:1
```

The other logs do not contain anything interesting sadly.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the job with `nomad agent -dev` and `consul agent -dev` on Fedora with SELinux enforcing, then inspect `var/log/audit/audit.log` and the Consul health-check logs. Done means the Envoy sidecar health checks become green without disabling SELinux or producing the reported socket denial.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
devops, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.