client `network_interface` fingerprinting incorrect/unintended IP address in `unique.network.ip-address`
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
```
Nomad v1.7.6
BuildDate 2024-03-12T07:27:36Z
Revision 594fedbfbc4f0e532b65e8a69b28ff9403eb822e
```
### Operating system and Environment details
Debian 11 on Linode (with Linode's Network Helper enabled)
### Issue
TLDR; Nomad client network_interface should allow for selection of a different addr resource within the same interface
- Nomad is hard-coded to select the first addr in client's network_interface (which is used to fingerprint for `unique.network.ip-address`).
- This behaviour renders nomad unusable for certain networking configurations. e.g. Linode with auto-networking helper enabled (default), which adds both public and private ip to the same eth0 interface.
- In such a situation, rather than hard-coded selection of the first `nwResources[0].IP` entry, which is the public IP, Nomad should let us select private IP's resource for the purpose of fingerprint for `unique.network.ip-address` from this interface.
- In other words, to fingerprint `unique.network.ip-address=$my_private_ip` in the above case, the client configuration should provide a way to choose which of the multiple available address resources from an interface should be used.
Related code:
```go
resp.AddAttribute("unique.network.ip-address", nwResources[0].IP)
```
Apparently the behaviour has been recognized as, "Deprecated, setting the first IP as unique IP for the node", but is yet to be worked upon:
https://github.com/hashicorp/nomad/blob/83720740f5a7f4053af2ba45dc687964de2a93cb/client/fingerprint/network.go#L111-L120
### Reproduction steps
Linode's automatic Network Helper tool sets up something like this:
```sh
$ cat /etc/network/interfaces
```
```haskell
# Generated by Linode Network Helper
# Sun May 12 14:16:06 2024 UTC
#
# This file is automatically generated on each boot with your Linode's
# current network configuration. If you need to modify this file, please
# first disable the 'Auto-configure networking' setting within your Linode's
# configuration profile:
# - https://cloud.linode.com/linodes/35915162/configurations
#
# For more information on Network Helper:
# - https://www.linode.com/docs/guides/network-helper/
#
# A backup of the previous config is at /etc/network/.interfaces.linode-last
# A backup of the original config is at /etc/network/.interfaces.linode-orig
#
# /etc/network/interfaces
auto lo
iface lo inet loopback
source /etc/network/interfaces.d/*
auto eth0
allow-hotplug eth0
iface eth0 inet6 auto
iface eth0 inet static
address 50.60.70.101/24
gateway 50.60.70.1
up ip addr add 192.168.120.225/17 dev eth0 label eth0:1
down ip addr del 192.168.120.225/17 dev eth0 label eth0:1
```
```sh
$ ip a
```
```haskell
3: eth0: mtu 1500 qdisc mq state UP group default qlen 1000
link/ether f2:3c:93:a1:9d:dd brd ff:ff:ff:ff:ff:ff
inet 50.60.70.101/24 brd 50.60.70.255 scope global eth0
valid_lft forever preferred_lft forever
inet 192.168.120.225/17 scope global eth0:1
valid_lft forever preferred_lft forever
inet6 2600:abcd:wxyz/64 scope global dynamic mngtmpaddr
valid_lft 5144sec preferred_lft 1544sec
inet6 fe80::abcd:wxyz/64 scope link
valid_lft forever preferred_lft forever
```
### Current behaviour
```hcl
client {
enabled = true
network_interface = "eth0"
}
```
```hcl
unique.network.ip-address: 50.60.70.101 # $my_public_ip, first from eth0. not useful
```
### Proposed behaviour
```hcl
client {
enabled = true
# allow label based interface addr resource selection
network_interface = "eth0:1"
# or, make it select the interface resource which matches CIDR
network_cidr = "192.168.120.225/17"
# with templating support
network_cidr = "{{ GetPrivateInterfaces | … | limit 1 | attr \"address\" }}"
}
```
```hcl
unique.network.ip-address: 192.168.120.225 # $my_private_ip from eth0:1
```
### Workaround
The only workaround to this that I've been able to come up with is setting up a dummy interface on the system. And then setting:
```hcl
client {
enabled = true
network_interface = "dummy10" # selects $my_private_ip
}
```
```haskell
14: dummy10: mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether 5f:1d:12:6d:a7:16 brd ff:ff:ff:ff:ff:ff
inet 192.168.120.225/17 brd 192.168.255.255 scope global dummy10
valid_lft forever preferred_lft forever
```
```hcl
unique.network.ip-address: 192.168.120.225 # $my_private_ip from dummy10
```
It works but comes with its own oddities. See https://github.com/hashicorp/nomad/issues/3675#issuecomment-504660287.
### Other considerations
- https://github.com/hashicorp/nomad/issues/3675#issuecomment-504660287
The issue with the workaround missed on fixing the use case. It introduced sockaddr templating to network_interface config.
- https://github.com/hashicorp/go-sockaddr
sockaddr templating does not help with this since network_interface only accepts interface names and not ip addresses or CIDRs.
- https://github.com/hashicorp/nomad/issues/19554
Apart from, say, `eth0`'s resource label like `eth0:1`, network interface names themselves can also contain colons in them. Perhaps the fix could first look for `eth0:1` interface, and then for `eth0:1` labelled resource within `eth0` interface, in that order.
- https://github.com/hashicorp/nomad/issues/11069
This seems related but with insufficient troubleshooting by the original poster?
- https://discuss.hashicorp.com/t/how-to-change-unique-network-ip-address-for-a-node/22696
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.