setting `port` in terminating gateway causes Envoy configuration to fail
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
v1.6.2
### Issue
Creating a Consul Terminating Gateway using the [gateway block](https://developer.hashicorp.com/nomad/docs/job-specification/gateway) fails to bind an envoy listener when in bridged network mode. Substituting `mode = "bridge"` with `mode = "host"` allows the end-to-end test to be successful.
### Reproduction steps
1. Register the external service
Catalog Registration Random Message
```sh
curl -X PUT -H "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
http://localhost:8500/v1/catalog/register -d '{
"Node": "external",
"Address": "whatthecommit.com",
"NodeMeta": {
"external-node": "true",
"external-probe": "true"
},
"Service": {
"Service": "random-message",
"Address": "whatthecommit.com",
"Port": 443
}
}'
```
3. Deploy the TGW
Job Spec
```sh
job "terminating-gateway" {
datacenters = ["*"]
group "terminating-gateway" {
network {
mode = "bridge"
port "envoy" {}
}
service {
name = "terminating-gateway"
port = "envoy"
connect {
gateway {
proxy {}
terminating {
service {
name = "random-message"
sni = "whatthecommit.com" # /index.txt
ca_file = "/etc/ssl/certs/ca-certificates.crt"
}
}
}
}
}
}
group "alpine" {
network {
mode = "bridge"
}
service {
name = "alpine"
connect {
sidecar_service {
proxy {
upstreams {
destination_name = "random-message"
local_bind_port = 8888
}
}
}
}
}
task "alpine" {
driver = "docker"
config {
image = "alpine:3.18.4"
command = "${NOMAD_TASK_DIR}/loop.sh"
}
template {
destination = "${NOMAD_TASK_DIR}/loop.sh"
perms = "755"
data = <<-EOF
#!/bin/sh
apk add curl jq iproute2-ss
while true ; do
sleep 86400
done
EOF
}
}
}
}
```
4. Create the service intention
Allow Alpine -> Random Message
```sh
curl -X PUT -H "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
http://localhost:8500/v1/config -d '{
"Kind": "service-intentions",
"Name": "random-message",
"Sources": [{
"Name": "alpine",
"Action": "allow"
}]
}'
```
6. Validation
`nomad alloc exec -task=alpine curl localhost:8888/index.txt`
#### Expected Result
`nomad alloc exec -task=alpine curl localhost:8888/index.txt` should produce a random commit message.
#### Actual Result
No listener has been created, but the allocations are in a running state. Consul agent logs lead me to infer that Nomad is attempting to pass-in a faulty configuration for Envoy's bootstrap configuration.
##### Consul Agent logs
```log
2023-10-17T23:04:06.222Z [ERROR] agent.envoy.xds.terminating_gateway: got error response from envoy proxy: service_id=_nomad-task-d747b1f2-8987-e2ff-2dc8-2b689f9fdcb7-group-terminating-gateway-terminating-gateway-listener typeUrl=type.googleapis.com/envoy.config.listener.v3.Listener xdsVersion=v3 nonce=00000002
error=
| rpc error: code = Internal desc = Proto constraint validation failed (ListenerValidationError.Address: embedded message failed validation | caused by AddressValidationError.SocketAddress: embedded message failed validation | caused by SocketAddressValidationError.PortValue: value must be less than or equal to 65535): name: "default:0.0.0.0:-1"
| address {
| socket_address {
| address: "0.0.0.0"
| port_value: 4294967295
| }
| }
```
##### Snippet from the catalog endpoint
Particularly the "Config" section
```json
"ServiceMeta": {
"external-source": "nomad"
},
"ServicePort": 21990,
"ServiceSocketPath": "",
"ServiceEnableTagOverride": false,
"ServiceProxy": {
"AccessLogs": {
"Enabled": true,
"Type": "stdout"
},
"Mode": "",
"Config": {
"connect_timeout_ms": 5000,
"envoy_gateway_bind_addresses": {
"default": {
"Address": "0.0.0.0",
"Port": -1
}
},
"envoy_gateway_no_default_bind": true,
"protocol": "http"
},
"MeshGateway": {
"Mode": "local"
},
"Expose": {}
```
Contributor guide
No contributing guide indexed for this repository
Research direction
No source file or test is named. Start by reproducing the terminating gateway job in bridged network mode and trace the gateway block through Nomad's generated Consul/Envoy configuration, focusing on envoy_gateway_bind_addresses. Done means the Envoy listener is created and the Alpine request to localhost:8888/index.txt returns a random commit message.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100