CNI-provided address is bound/advertised correctly, wrong in Nomad API and env vars
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
Nomad v1.2.6 (95514d569610f15ce49b4a7a1a6bfd3e7b3e7b4f)
### Operating system and Environment details
Gentoo server, running a test cluster in docker containers (consul and nomad server are in docker, nomad client is running on the host)
### Issue
I am experiencing a very similar issue to #11216. I have a CNI macvlan network defined, and a nomad job using this network.
The IP-related environment variables in the container get set to the host address (10.17.17.1), and not the container address (10.17.17.X)
```
"NOMAD_ADDR_inbound":"10.17.17.1:22638"
"NOMAD_HOST_ADDR_inbound":"10.17.17.1:22638"
"NOMAD_HOST_IP_inbound":"10.17.17.1"
"NOMAD_IP_inbound":"10.17.17.1"
```
The alloc's Allocation Address is also incorrect:
```
$ nomad alloc status 30e443d0
...
Allocation Addresses (mode = "cni/ingress")
Label Dynamic Address
*inbound yes 10.17.17.1:22638
...
```
The ServiceAddresses are populated correctly in consul, though:
```
$ curl -s 10.0.150.200:8500/v1/catalog/service/demo-cni | jq | grep ServiceAddress
"ServiceAddress": "10.17.17.36",
"ServiceAddress": "10.17.17.35",
"ServiceAddress": "10.17.17.34",
```
### Reproduction steps
Create a CNI network as follows:
```
$ cat /opt/cni/config/ingress.conflist
{
"cniVersion": "0.4.0",
"name": "ingress",
"plugins": [
{
"type": "macvlan",
"master": "bond0",
"ipam": {
"type": "host-local",
"ranges": [
[
{
"subnet": "10.17.17.0/24",
"rangeStart": "10.17.17.32",
"rangeEnd": "10.17.17.40",
"gateway": "10.17.17.254"
}
]
]
}
}
]
}
```
Start a job as follows:
```
$ cat demo-cni.hcl
job "demo-cni" {
datacenters = ["dc1"]
group "demo_cni" {
count = 3
network {
mode = "cni/ingress"
port "inbound" { }
}
service {
name = "demo-cni"
port = "inbound"
address_mode = "alloc"
tags = [
"traefik.enable=true",
"traefik.http.routers.test_cni.rule=Path(`/testcni`)",
]
}
task "server" {
env {
PORT = "${NOMAD_PORT_inbound}"
}
driver = "docker"
config {
image = "ealen/echo-server"
ports = ["inbound"]
}
}
}
}
```
#### Expected Result
Environment variables and Allocation Addresses are set to the container address
#### Actual Result
Environment variables and Allocation Addresses are set to the host address
### Job file (if appropriate)
See above
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the issue with /opt/cni/config/ingress.conflist and demo-cni.hcl, then compare nomad alloc status output and the container's NOMAD_* variables with the Consul catalog's ServiceAddress values. Trace how the CNI-assigned address differs from the host address; done means Allocation Addresses and environment variables use the container address while service registration remains correct.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go
- Domain
- backend, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100