ServiceAddress in Consul isn't updated when Nomad's client_interface changes
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
Hello 👋
I run Nomad in a configuration where I occasionally need to change the Nomad `client_interface`, without requiring any downtime for the services under Nomad's control. The purpose of changing the `client_interface` is to influence the address advertised for these services in Consul.
The system in question undergoes occasional changes where the network interface and address used for these service advertisements needs to be updated without an interruption to the underlying services.
I have observed that the registrations in Consul aren't updated to reflect this change unless I completely stop, and re-run the relevant jobs, leading to clients continuing to try and contact these services via the address attached to the previous `client_interface`, rather than the updated one.
Is this the intended behaviour? Is there any way to update these Consul service registrations without interrupting these services or registrations?
Apologies if this is a duplicate of an existing issue. I did locate https://github.com/hashicorp/nomad/issues/4815, which is related, and would possibly be a suitable workaround if it was available.
Happy to provide further information and background on this use-case and how to replicate this if needed! Thanks!
### Nomad version
```
vagrant@node-1:~$ nomad version
Nomad v0.12.3 (2db8abd9620dd41cb7bfe399551ba0f7824b3f61)
vagrant@node-1:~$ consul version
Consul v1.8.3
Revision a9322b9c7
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)
```
### Operating system and Environment details
Debian GNU/Linux 10 (buster) AMD64 running in VirtualBox via Vagrant.
### Issue
The `ServiceAddress` and `ServiceTaggedAddresses` for the Service registration in Consul, as well as the address used for associated Checks are not updated in Consul when `network_interface` is updated in Nomad's client configuration.
### Reproduction steps
1. Deploy a Consul integrated Nomad instance with two Ethernet interfaces (`eth0` and `eth1`) with unique addresses. Start Nomad with `client_interface` set to `eth0`:
```
client {
enabled = true
network_interface = "eth0"
}
```
2. Run a new job which contains at least one service and check.
3. Confirm that the `ServiceAddress` and `ServiceTaggedAddresses` for the Service registration in Consul, as well as the address used for associated Check(s) in the Consul registration reflect the IP address of `eth0`.
4. Update the Nomad configuration to `eth1`:
```
client {
enabled = true
network_interface = "eth1"
}
```
5. Restart Nomad.
6. Observe that the `ServiceAddress` and `ServiceTaggedAddresses` for the Service registration in Consul, as well as the address used for associated Check(s) in the Consul registration continue to reflect the IP address of `eth0`.
7. Re-run the already allocated job.
8. Observe that these have still not updated.
9. Stop and re-run the job.
10. Observe that these have updated.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with Nomad's client configuration handling for network_interface and the Consul service-registration path, then follow the reproduction steps using two interfaces. Done means ServiceAddress, ServiceTaggedAddresses, and associated check addresses update after the interface changes without stopping or re-running the job.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100