hashicorp / hashicorp/nomad

setting `port` in terminating gateway causes Envoy configuration to fail

Open
#18,802 3 comments 0 reactions 0 assignees View on GitHub
theme/consul theme/consul/connect type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
v1.6.2

### Issue
Creating a Consul Terminating Gateway using the [gateway block](https://developer.hashicorp.com/nomad/docs/job-specification/gateway) fails to bind an envoy listener when in bridged network mode. Substituting `mode = "bridge"` with `mode = "host"` allows the end-to-end test to be successful.

### Reproduction steps
1. Register the external service

Catalog Registration Random Message

```sh
curl -X PUT -H "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
http://localhost:8500/v1/catalog/register -d '{
"Node": "external",
"Address": "whatthecommit.com",
"NodeMeta": {
"external-node": "true",
"external-probe": "true"
},
"Service": {
"Service": "random-message",
"Address": "whatthecommit.com",
"Port": 443
}
}'
```

3. Deploy the TGW

Job Spec

```sh
job "terminating-gateway" {
datacenters = ["*"]

group "terminating-gateway" {

network {
mode = "bridge"
port "envoy" {}
}

service {
name = "terminating-gateway"
port = "envoy"
connect {
gateway {
proxy {}
terminating {
service {
name = "random-message"
sni = "whatthecommit.com" # /index.txt
ca_file = "/etc/ssl/certs/ca-certificates.crt"
}
}
}
}
}
}


group "alpine" {
network {
mode = "bridge"
}

service {
name = "alpine"
connect {
sidecar_service {
proxy {
upstreams {
destination_name = "random-message"
local_bind_port = 8888
}
}
}
}
}

task "alpine" {
driver = "docker"
config {
image = "alpine:3.18.4"
command = "${NOMAD_TASK_DIR}/loop.sh"
}

template {
destination = "${NOMAD_TASK_DIR}/loop.sh"
perms = "755"
data = <<-EOF
#!/bin/sh
apk add curl jq iproute2-ss

while true ; do
sleep 86400
done
EOF
}
}
}
}
```

4. Create the service intention

Allow Alpine -> Random Message

```sh
curl -X PUT -H "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
http://localhost:8500/v1/config -d '{
"Kind": "service-intentions",
"Name": "random-message",
"Sources": [{
"Name": "alpine",
"Action": "allow"
}]
}'
```

6. Validation
`nomad alloc exec -task=alpine curl localhost:8888/index.txt`

#### Expected Result
`nomad alloc exec -task=alpine curl localhost:8888/index.txt` should produce a random commit message.

#### Actual Result
No listener has been created, but the allocations are in a running state. Consul agent logs lead me to infer that Nomad is attempting to pass-in a faulty configuration for Envoy's bootstrap configuration.

##### Consul Agent logs

```log
2023-10-17T23:04:06.222Z [ERROR] agent.envoy.xds.terminating_gateway: got error response from envoy proxy: service_id=_nomad-task-d747b1f2-8987-e2ff-2dc8-2b689f9fdcb7-group-terminating-gateway-terminating-gateway-listener typeUrl=type.googleapis.com/envoy.config.listener.v3.Listener xdsVersion=v3 nonce=00000002
error=
| rpc error: code = Internal desc = Proto constraint validation failed (ListenerValidationError.Address: embedded message failed validation | caused by AddressValidationError.SocketAddress: embedded message failed validation | caused by SocketAddressValidationError.PortValue: value must be less than or equal to 65535): name: "default:0.0.0.0:-1"
| address {
| socket_address {
| address: "0.0.0.0"
| port_value: 4294967295
| }
| }
```

##### Snippet from the catalog endpoint
Particularly the "Config" section

```json
"ServiceMeta": {
"external-source": "nomad"
},
"ServicePort": 21990,
"ServiceSocketPath": "",
"ServiceEnableTagOverride": false,
"ServiceProxy": {
"AccessLogs": {
"Enabled": true,
"Type": "stdout"
},
"Mode": "",
"Config": {
"connect_timeout_ms": 5000,
"envoy_gateway_bind_addresses": {
"default": {
"Address": "0.0.0.0",
"Port": -1
}
},
"envoy_gateway_no_default_bind": true,
"protocol": "http"
},
"MeshGateway": {
"Mode": "local"
},
"Expose": {}
```

Contributor guide

No contributing guide indexed for this repository

Research direction

No source file or test is named. Start by reproducing the terminating gateway job in bridged network mode and trace the gateway block through Nomad's generated Consul/Envoy configuration, focusing on envoy_gateway_bind_addresses. Done means the Envoy listener is created and the Alpine request to localhost:8888/index.txt returns a random commit message.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.