hashicorp / hashicorp/nomad

Service checks fails when exposing two paths for the same port in "consul connect"-enabled service (with transparent_proxy)

Open
#25,262 1 comment 0 reactions 0 assignees View on GitHub
hcc/jira stage/needs-investigation theme/consul theme/consul/connect type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version (client)
Nomad v1.9.6
BuildDate 2025-02-11T18:55:10Z
Revision 7f8b44963d36d025520348d7f24735774d26f13b+CHANGES

### Nomad version (server)
Nomad v1.9.5
BuildDate 2025-01-14T18:35:12Z
Revision 0b7bb8b60758981dae2a78a0946742e09f8316f5+CHANGES

### Issue
In Nomad, to make `checks` work with Consul Connect, you need to use the `export` stanza (or set `export = true` in checks). From my tests, this doesn't actually fully expose the endpoint, it still goes through the Envoy sidecar, but it bypasses the need for mutual TLS and namespace isolation. This lets other services (like Consul healthcheck) reach the endpoint without being in the Consul service mesh.

I've been using this mechanism to get Consul healthcheck working with my services, and according to the [documentation](https://developer.hashicorp.com/nomad/docs/job-specification/expose#path-examples), you can expose two different paths (even with different protocols) on the same port in a Consul Connect–enabled service (with `transparent_proxy`).

That's when the issue occurs: on the Envoy sidecar, the task hook fails to bootstrap and throws an error without any details:

```
Task hook failed: envoy_bootstrap: error creating bootstrap configuration for Connect proxy sidecar: exit status 1; see:
```

Note that this only happens when you're exposing two different paths using [`proxy.expose.path`](https://developer.hashicorp.com/nomad/docs/job-specification/expose) as shown in my job file below.

It's weird because the issue seems to occur more often when the service is updating or recycling replicas. I was able to reproduce it when creating a job, but it feels inconsistent, almost like it might be a race condition.

### Reproduction steps
2. Run the job with the definition listed at bottom of this issue:
$ `nomad job run healthcheck.nomad.hcl`

#### Expected Result
I expect Envoy to bootstrap correctly and expose both paths on the same port, as described in the [documentation](https://developer.hashicorp.com/nomad/docs/job-specification/expose#path-examples).

#### Actual Result
Envoy fails to bootstrap due to an error in its configuration. I don't have access to the generated configuration right now, so I can't provide further details.

### Job file
```
job "healthcheck" {
datacenters = ["dc1"]

group "hashicorp" {
network {
mode = "bridge"

port "healthcheck-http" {
to = 5678
}
}

service {
connect {
sidecar_service {
proxy {
transparent_proxy {}

expose {
path {
path = "/health"
protocol = "http"
local_path_port = 5678
listener_port = "healthcheck-http"
}

path {
path = "/ready"
protocol = "http"
local_path_port = 5678
listener_port = "healthcheck-http"
}
}
}
}
}

# Liveness
check {
type = "http"
path = "/health"
port = "healthcheck-http"
interval = "1s"
timeout = "3s"

check_restart {
limit = 3
grace = "5s"
}
}

# Readiness
check {
type = "http"
path = "/ready"
port = "healthcheck-http"
interval = "5s"
timeout = "3s"
}
}

task "echo" {
driver = "docker"

config {
image = "docker.io/hashicorp/http-echo:latest"
args = [
"-text=Hi!"
]
}
}
}
}
```

### Extras
I've reproduced this problem even with a "naked" Consul, without any complex service configurations or additional settings. I'm also not sure whether this issue should be reported to the Nomad or Consul repository, as it appears to be more of a Consul Connect issue rather than a Nomad one.

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the failure with healthcheck.nomad.hcl using two proxy.expose.path entries on the same listener port and inspect the Envoy bootstrap error. Compare the generated configuration with the expected behavior: Envoy should bootstrap successfully and expose both /health and /ready paths.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.