hashicorp / hashicorp/nomad

Nomad does not re-create connect-proxy docker containers on envoy version changes

Open
#19,095 6 comments 0 reactions 0 assignees View on GitHub
stage/accepted stage/needs-discussion theme/consul/connect type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
```
$ nomad version
Nomad v1.6.1
BuildDate 2023-07-21T13:49:42Z
Revision 515895c7690cdc72278018dc5dc58aca41204ccc
```

```
$ consul version
Consul v1.16.2
Revision 68f81912
Build Date 2023-09-19T19:29:18Z
```

### Operating system and Environment details

```
$ cat /etc/*release*
CentOS Linux release 7.9.2009 (Core)
```

### Issue

We did an upgrade of the Consul version on the cluster. At the same time, we upgraded the Envoy version to the compatible one. After the components were upgraded and both consul and nomad clusters restarted we did the product rollout. We use consul-connect. If the product version was changed - all was good, the new Envoy was delivered. In the case when the product version was the same - Envoy was also the old one.

### Reproduction steps

```
$ /opt/nomad/current/nomad init -connect
$ /opt/nomad/current/nomad job run example.nomad.hcl

# compare the Envoy versions
$ sudo cat /opt/nomad/conf.d/nomad.hcl | grep envoy
$ docker ps | grep envoy

# update the Envoy version in the nomad.hcl file, do nomad restart

$ sudo vim /opt/nomad/conf.d/nomad.hcl
$ sudo cat /opt/nomad/conf.d/nomad.hcl | grep envoy

$ /opt/nomad/current/nomad job run example.nomad.hcl
$ docker ps | grep envoy

# As a result, the Envoy version will be the same as it was before the upgrade

```

#### Expected Result

The Envoy version be updated

#### Actual Result

The Envoy version stays the same.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the connect setup from `nomad init -connect` and the job in `example.nomad.hcl`; compare the Envoy image configuration in `nomad.hcl` before and after restarting Nomad. Trace why re-running the unchanged product job does not replace the Docker proxy container after the Envoy version changes. Done means the proxy uses the updated Envoy version without requiring a product version change.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.