hashicorp / hashicorp/nomad

Consul Connect service health checks not accessible?

Open
#9,907 11 comments 6 reactions 0 assignees View on GitHub
theme/consul theme/consul/connect theme/service-discovery/consul type/enhancement
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version

`Nomad v1.0.2 (4c1d4fc6a5823ebc8c3e748daec7b4fda3f11037)`

### Operating system and Environment details

Ubuntu 18.04

### Issue

When running a service binding a port locally (ex `127.0.0.1:8080`), it seems that Consul health checks cannot access them, and I'm unable to use options like `expose` or `address_mode`.

I would expect this to be a pretty common approach if I understand correctly (to avoid leaking ports that could be accessed outside of Consul Connect). Can the guides/docs add steps for health checks in https://www.nomadproject.io/docs/integrations/consul-connect?

### Reproduction steps

Using the following job, try adding `expose = true` or `address_mode = "driver"` to the check and note the errors.

**With `expose = true`**:

```
❯ nomad job run debug/python_http.hcl
Error submitting job: Unexpected response code: 500 (error in job mutator expose-check: unable to determine local service port for service check app->python-http->python-http-health)
```

This happens even if I pass `port = "8080"` in the `check` configuration.

**With `address_mode = "driver"`**:

The job is deployed, but the task fails with the following log:

```
failed to setup alloc: pre-run hook "group_services" failed: error getting address for check "python-http-health": cannot use address_mode="driver": no driver network exists
```

### Job file (if appropriate)

```hcl
job "python-http" {
datacenters = ["kitchen"]

group "app" {
network {
mode = "bridge"
port "http" {}
}

task "python-http" {
driver = "docker"

config {
image = "python:3"
command = "python3"
args = [
"-m",
"http.server",
"-b",
"127.0.0.1",
"${NOMAD_PORT_http}",
]
}

env {
PYTHONUNBUFFERED = "1"
}

resources {
cpu = 20
memory = 100
}
}

service {
name = "python-http"
port = "http"

check {
type = "http"
name = "python-http-health"
path = "/"
interval = "10s"
timeout = "3s"
# address_mode = "driver"
# expose = "true"
}

connect {
sidecar_service {}
}
}
}
}
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the Consul Connect integration guide at https://www.nomadproject.io/docs/integrations/consul-connect and review the supplied Python job file and health-check errors. Reproduce the `expose` and `address_mode` cases, then document the steps and configuration needed for health checks on locally bound service ports, including the expected behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
documentation, networking
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.