hashicorp / hashicorp/nomad

ServiceAddress in Consul isn't updated when Nomad's client_interface changes

Open
#8,732 4 comments 0 reactions 0 assignees View on GitHub
stage/accepted theme/consul theme/service-discovery/consul type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

Hello 👋

I run Nomad in a configuration where I occasionally need to change the Nomad `client_interface`, without requiring any downtime for the services under Nomad's control. The purpose of changing the `client_interface` is to influence the address advertised for these services in Consul.

The system in question undergoes occasional changes where the network interface and address used for these service advertisements needs to be updated without an interruption to the underlying services.

I have observed that the registrations in Consul aren't updated to reflect this change unless I completely stop, and re-run the relevant jobs, leading to clients continuing to try and contact these services via the address attached to the previous `client_interface`, rather than the updated one.

Is this the intended behaviour? Is there any way to update these Consul service registrations without interrupting these services or registrations?

Apologies if this is a duplicate of an existing issue. I did locate https://github.com/hashicorp/nomad/issues/4815, which is related, and would possibly be a suitable workaround if it was available.

Happy to provide further information and background on this use-case and how to replicate this if needed! Thanks!

### Nomad version
```
vagrant@node-1:~$ nomad version
Nomad v0.12.3 (2db8abd9620dd41cb7bfe399551ba0f7824b3f61)
vagrant@node-1:~$ consul version
Consul v1.8.3
Revision a9322b9c7
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)
```

### Operating system and Environment details

Debian GNU/Linux 10 (buster) AMD64 running in VirtualBox via Vagrant.

### Issue

The `ServiceAddress` and `ServiceTaggedAddresses` for the Service registration in Consul, as well as the address used for associated Checks are not updated in Consul when `network_interface` is updated in Nomad's client configuration.

### Reproduction steps

1. Deploy a Consul integrated Nomad instance with two Ethernet interfaces (`eth0` and `eth1`) with unique addresses. Start Nomad with `client_interface` set to `eth0`:

```
client {
enabled = true
network_interface = "eth0"
}
```

2. Run a new job which contains at least one service and check.

3. Confirm that the `ServiceAddress` and `ServiceTaggedAddresses` for the Service registration in Consul, as well as the address used for associated Check(s) in the Consul registration reflect the IP address of `eth0`.

4. Update the Nomad configuration to `eth1`:

```
client {
enabled = true
network_interface = "eth1"
}
```

5. Restart Nomad.

6. Observe that the `ServiceAddress` and `ServiceTaggedAddresses` for the Service registration in Consul, as well as the address used for associated Check(s) in the Consul registration continue to reflect the IP address of `eth0`.

7. Re-run the already allocated job.

8. Observe that these have still not updated.

9. Stop and re-run the job.

10. Observe that these have updated.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with Nomad's client configuration handling for network_interface and the Consul service-registration path, then follow the reproduction steps using two interfaces. Done means ServiceAddress, ServiceTaggedAddresses, and associated check addresses update after the interface changes without stopping or re-running the job.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.