hashicorp / hashicorp/nomad

Cannot change ingress container from http to tcp (or vice versa) when using Consul Service Mesh

Open
#14,802 3 comments 0 reactions 0 assignees View on GitHub
stage/accepted theme/consul/connect theme/networking type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
Nomad v1.3.5 (1359c2580fed080295840fb888e28f0855e42d50)

### Operating system and Environment details
Ubuntu 22.04 on AWS (on a fresh EC2 instance), amd64

Consul v1.13.2
Revision 0e046bbb
Build Date 2022-09-20T20:30:07Z
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

Docker version 20.10.18, build b40c2f6

### Issue

If I run an ingress container with the `http` protocol, I'm unable to edit it to use `tcp` even after I stop the job. Even if I run `nomad system gc` and `nomad system reconcile summaries`, it still doesn't work. I'm also unable to edit the consul config to use

If I swap all instances of `http` and `tcp` I get the same errors.

### Reproduction steps
1. Start nomad/consul in dev mode:
```
consul agent -dev
sudo nomad agent -dev-connect
```

2. Set up consul to use http as default protocol (using proxy-defaults.hcl file below)
```
consul config write proxy-defaults.hcl
```

3. Run the first job file
```
nomad job run job1.nomad
```

4. After job has started, stop the job
```
nomad job stop job1
```

5. When job stops successfully, run the second job file
```
nomad job run job2.nomad
```

#### Expected Result
I should be able to run job2 as normal.

#### Actual Result
```
$ nomad job run job2.nomad
Error submitting job: Unexpected response code: 500 (Unexpected response code: 500 (service "test-upstream" has protocol "http", which does not match defined listener protocol "tcp"))
$ consul config write service-defaults.hcl
Error writing config entry service-defaults/test-upstream: Unexpected response code: 500 (service "test-upstream" has protocol "tcp", which does not match defined listener protocol "http")
```

### Job file (if appropriate)

proxy-defaults.hcl
```
Kind = "proxy-defaults"
Name = "global"
Config {
protocol = "http"
}
```

service-defaults.hcl
```
Kind = "service-defaults"
Name = "test-upstream"
Protocol = "tcp"
```

job1.nomad:
```
job "job1" {
region = "global"
datacenters = ["dc1"]
type = "system"

group "group1" {
network {
mode = "bridge"
port "default" {
static = 12345
to = 12345
}
}
service {
name = "test-ingress"
port = "12345"
connect {
gateway {
proxy {
connect_timeout = "5s"
}
ingress {
listener {
port = 12345
protocol = "http"
service {
name = "test-upstream"
hosts = ["*"]
}
}
}
}
}
}
}
}
```

job2.nomad:
```
job "job2" {
region = "global"
datacenters = ["dc1"]
type = "system"

group "group2" {
network {
mode = "bridge"
port "default" {
static = 12345
to = 12345
}
}
service {
name = "test-ingress"
port = "12345"
connect {
gateway {
proxy {
connect_timeout = "5s"
}
ingress {
listener {
port = 12345
protocol = "tcp"
service {
name = "test-upstream"
}
}
}
}
}
}
}
}
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by running the listed reproduction with proxy-defaults.hcl, service-defaults.hcl, job1.nomad, and job2.nomad in the Nomad and Consul dev agents. Trace what remains after stopping job1 and then compare the ingress listener and service protocol state when job2 is submitted. Done means switching the listener between http and tcp succeeds and the corresponding Consul configuration can be written.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker
Domain
infrastructure, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.