hashicorp / hashicorp/nomad

validate against having multiple `expose.path` to same port

Open
#10,841 5 comments 0 reactions 0 assignees View on GitHub
hcc/jira stage/accepted theme/consul/connect type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
nomad v1.1.2

### Operating system and Environment details
Debian 10

### Issue
When exposing multiple paths using the following job file:
```hcl
job "expose-example" {
datacenters = ["dc1"]

group "api" {
network {
mode = "bridge"

port "api_expose_healthcheck" {
to = -1
}
}

service {
name = "count-api"
port = "9001"

connect {
sidecar_service {
proxy {
expose {
path {
path = "/health"
protocol = "http"
local_path_port = 9001
listener_port = "api_expose_healthcheck"
}
path {
path = "/test"
protocol = "http"
local_path_port = 9001
listener_port = "api_expose_healthcheck"
}
}
}
}
}

check {
name = "api-health"
type = "http"
path = "/health"
port = "api_expose_healthcheck"
interval = "10s"
timeout = "3s"
}
}

task "web" {
driver = "docker"

config {
image = "hashicorpnomad/counter-api:v3"
}

# e.g. reference ${NOMAD_PORT_api_expose_healthcheck} for other uses
}
}
}
```

You can only access the first path specified, but the second returns a 404:
```
$ root@do-sfo3-s-1vcpu-2gb-amd-nomad-client-0:~# curl -v 10.124.0.9:25995/health
* Expire in 0 ms for 6 (transfer 0x555fbf620fb0)
* Trying 10.124.0.9...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x555fbf620fb0)
* Connected to 10.124.0.9 (10.124.0.9) port 25995 (#0)
> GET /health HTTP/1.1
> Host: 10.124.0.9:25995
> User-Agent: curl/7.64.0
> Accept: */*
>
< HTTP/1.1 200 OK
< date: Fri, 02 Jul 2021 05:28:27 GMT
< content-length: 26
< content-type: text/plain; charset=utf-8
< x-envoy-upstream-service-time: 6
< server: envoy
<
Hello, you've hit /health
* Connection #0 to host 10.124.0.9 left intact

$ root@do-sfo3-s-1vcpu-2gb-amd-nomad-client-0:~# curl -v 10.124.0.9:25995/test
* Expire in 0 ms for 6 (transfer 0x562d38804fb0)
* Trying 10.124.0.9...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x562d38804fb0)
* Connected to 10.124.0.9 (10.124.0.9) port 25995 (#0)
> GET /test HTTP/1.1
> Host: 10.124.0.9:25995
> User-Agent: curl/7.64.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< date: Fri, 02 Jul 2021 05:28:30 GMT
< server: envoy
< content-length: 0
<
* Connection #0 to host 10.124.0.9 left intact
```

Looking at the envoy proxy logs it looks like both are being enabled:
![image](https://user-images.githubusercontent.com/17842707/124225487-5f268980-dabc-11eb-910f-3d182e0b8dd3.png)

If you flip the order, in the job file i.e:
```hcl
connect {
sidecar_service {
proxy {
expose {
path {
path = "/health"
protocol = "http"
local_path_port = 9001
listener_port = "api_expose_healthcheck"
}
path {
path = "/test"
protocol = "http"
local_path_port = 9001
listener_port = "api_expose_healthcheck"
}
}
}
}
}
```

You can then curl `/health` but not `/test`.

Looking at the envoy debug log, it shows "no cluster match for URL '/test'" and when I change the paths to use two different ports:
```hcl
sidecar_service {
proxy {
expose {
path {
path = "/health"
protocol = "http"
local_path_port = 9001
listener_port = "api_expose_healthcheck"
}
path {
path = "/test"
protocol = "http"
local_path_port = 9001
listener_port = "api_expose_healthcheck2"
}
}
}
}
}
```

it starts to work.

### Expected Outcome
Judging by the [expose documentation](https://www.nomadproject.io/docs/job-specification/expose#path-examples) you should be able to specify multiple exposed paths for a port; however, you may not be able to do it for the same protocol? I am also not sure if this is something wrong with:
+ my configuration
+ consul connect
+ nomad
+ envoy

causing this behavior, but wanted to start here incase it is my configuration or something known.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the expose.path documentation and reproduce the supplied HCL job using two paths on the same listener port, then compare it with the working two-port configuration. Inspect the Nomad, Consul Connect, and Envoy behavior described in the issue; done should establish whether duplicate listener ports are supported or rejected with clear validation.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.