Cannot change ingress container from http to tcp (or vice versa) when using Consul Service Mesh
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
Nomad v1.3.5 (1359c2580fed080295840fb888e28f0855e42d50)
### Operating system and Environment details
Ubuntu 22.04 on AWS (on a fresh EC2 instance), amd64
Consul v1.13.2
Revision 0e046bbb
Build Date 2022-09-20T20:30:07Z
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)
Docker version 20.10.18, build b40c2f6
### Issue
If I run an ingress container with the `http` protocol, I'm unable to edit it to use `tcp` even after I stop the job. Even if I run `nomad system gc` and `nomad system reconcile summaries`, it still doesn't work. I'm also unable to edit the consul config to use
If I swap all instances of `http` and `tcp` I get the same errors.
### Reproduction steps
1. Start nomad/consul in dev mode:
```
consul agent -dev
sudo nomad agent -dev-connect
```
2. Set up consul to use http as default protocol (using proxy-defaults.hcl file below)
```
consul config write proxy-defaults.hcl
```
3. Run the first job file
```
nomad job run job1.nomad
```
4. After job has started, stop the job
```
nomad job stop job1
```
5. When job stops successfully, run the second job file
```
nomad job run job2.nomad
```
#### Expected Result
I should be able to run job2 as normal.
#### Actual Result
```
$ nomad job run job2.nomad
Error submitting job: Unexpected response code: 500 (Unexpected response code: 500 (service "test-upstream" has protocol "http", which does not match defined listener protocol "tcp"))
$ consul config write service-defaults.hcl
Error writing config entry service-defaults/test-upstream: Unexpected response code: 500 (service "test-upstream" has protocol "tcp", which does not match defined listener protocol "http")
```
### Job file (if appropriate)
proxy-defaults.hcl
```
Kind = "proxy-defaults"
Name = "global"
Config {
protocol = "http"
}
```
service-defaults.hcl
```
Kind = "service-defaults"
Name = "test-upstream"
Protocol = "tcp"
```
job1.nomad:
```
job "job1" {
region = "global"
datacenters = ["dc1"]
type = "system"
group "group1" {
network {
mode = "bridge"
port "default" {
static = 12345
to = 12345
}
}
service {
name = "test-ingress"
port = "12345"
connect {
gateway {
proxy {
connect_timeout = "5s"
}
ingress {
listener {
port = 12345
protocol = "http"
service {
name = "test-upstream"
hosts = ["*"]
}
}
}
}
}
}
}
}
```
job2.nomad:
```
job "job2" {
region = "global"
datacenters = ["dc1"]
type = "system"
group "group2" {
network {
mode = "bridge"
port "default" {
static = 12345
to = 12345
}
}
service {
name = "test-ingress"
port = "12345"
connect {
gateway {
proxy {
connect_timeout = "5s"
}
ingress {
listener {
port = 12345
protocol = "tcp"
service {
name = "test-upstream"
}
}
}
}
}
}
}
}
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by running the listed reproduction with proxy-defaults.hcl, service-defaults.hcl, job1.nomad, and job2.nomad in the Nomad and Consul dev agents. Trace what remains after stopping job1 and then compare the ingress listener and service protocol state when job2 is submitted. Done means switching the listener between http and tcp succeeds and the corresponding Consul configuration can be written.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker
- Domain
- infrastructure, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100