hashicorp / hashicorp/nomad

Nomad does not expose cni issued ip address

Open
#16,779 6 comments 1 reaction 0 assignees View on GitHub
hcc/cst stage/needs-investigation theme/networking type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
```nomad --version
Nomad v1.5.2
BuildDate 2023-03-21T22:54:38Z
Revision 9a2fdb5f53dce81edf2802f0b64962e07596fd03
```

### Operating system and Environment details
```# cat /etc/issue
Ubuntu 22.10 \n \l
```
### Issue
When using cni, nomad does not show the allocation's ip address anywhere in the nomad cli/ui
in fact the only way to know what the ip address of an allocation is, is to look within the container itself (often most containers do not have binaries in them to look at ip details etc

### Reproduction steps
configure cni:
```
$ cat frontend.conflist
{
"cniVersion": "0.4.0",
"name": "frontend",
"args": { "cni": { "ips": ["10.0.10.128/24"] }},
"plugins": [
{
"type": "macvlan",
"master": "bond0.11",
"ipam": {
"type": "host-local",
"dataDir": "/var/run/nomad/cni-frontend-ipam-state",
"subnet": "10.0.10.0/24",
"rangeStart": "10.0.10.2",
"rangeEnd": "10.0.10.254",
"gateway": "10.0.10.1",
"routes": [
{ "dst": "0.0.0.0/0" },
{ "dst": "169.254.1.1/32", "gw": "10.0.10.1"}
]
}
},
{ "type": "portmap", "snat": true, "capabilities": { "portMappings": true } }
]
}
```

configure nomad job:
```
job "nginx" {
datacenters = ["dc1"]
meta {
deploy = "1"
}

group "nginx" {
count = "1"
network {
mode = "cni/frontend"
}
task "nginx" {
driver = "docker"
config {
image = "nginx:latest"
}
}
service {
name = "www"
port = 80
address_mode = "alloc"
}
}
}
```

plan and run job:

```
$ nomad job plan nginx.nomad
+ Job: "nginx"
+ Task Group: "nginx" (1 create)
+ Task: "nginx" (forces create)

Scheduler dry-run:
- All tasks successfully allocated.

Job Modify Index: 0
To submit the job with version verification run:

nomad job run -check-index 0 nginx.nomad

When running the job with the check-index flag, the job will only be run if the
job modify index given matches the server-side version. If the index has
changed, another user has modified the job and the plan's results are
potentially invalid.
```

```
$ nomad job run -check-index 0 nginx.nomad
==> 2023-04-04T08:17:38+10:00: Monitoring evaluation "4e42a013"
2023-04-04T08:17:39+10:00: Evaluation triggered by job "nginx"
2023-04-04T08:17:39+10:00: Evaluation within deployment: "020affbe"
2023-04-04T08:17:39+10:00: Allocation "2ff40152" created: node "37df7b89", group "nginx"
2023-04-04T08:17:39+10:00: Evaluation status changed: "pending" -> "complete"
==> 2023-04-04T08:17:39+10:00: Evaluation "4e42a013" finished with status "complete"
==> 2023-04-04T08:17:39+10:00: Monitoring deployment "020affbe"
✓ Deployment "020affbe" successful

2023-04-04T08:17:58+10:00
ID = 020affbe
Job ID = nginx
Job Version = 0
Status = successful
Description = Deployment completed successfully

Deployed
Task Group Desired Placed Healthy Unhealthy Progress Deadline
nginx 1 1 1 0 2023-04-03T17:24:42Z
```

get the status of job and allocation:
```
$ nomad job status nginx
ID = nginx
Name = nginx
Submit Date = 2023-04-04T03:13:09+10:00
Type = service
Priority = 50
Datacenters = dc1
Namespace = default
Status = running
Periodic = false
Parameterized = false

Summary
Task Group Queued Starting Running Failed Complete Lost Unknown
nginx 0 0 1 0 0 0 0

Latest Deployment
ID = 020affbe
Status = successful
Description = Deployment completed successfully

Deployed
Task Group Desired Placed Healthy Unhealthy Progress Deadline
nginx 1 1 1 0 2023-04-03T17:24:42Z

Allocations
ID Node ID Task Group Version Desired Status Created Modified
2ff40152 37df7b89 nginx 0 run running 5h7m ago 5h6m ago
```

```
$ nomad alloc status 2ff40152
ID = 2ff40152-c4b4-8ce8-9c7e-09ec416a9f88
Eval ID = 4e42a013
Name = nginx.nginx[0]
Node ID = 37df7b89
Node Name = frontend-1-1
Job ID = nginx
Job Version = 0
Client Status = running
Client Description = Tasks are running
Desired Status = run
Desired Description =
Created = 5h7m ago
Modified = 5h7m ago
Deployment ID = 020affbe
Deployment Health = healthy

Task "nginx" is "running"
Task Resources:
CPU Memory Disk Addresses
0/100 MHz 76 MiB/300 MiB 300 MiB

Task Events:
Started At = 2023-04-03T17:14:32Z
Finished At = N/A
Total Restarts = 0
Last Restart = N/A

Recent Events:
Time Type Description
2023-04-04T03:14:32+10:00 Started Task started by client
2023-04-04T03:14:25+10:00 Driver Downloading image
2023-04-04T03:14:25+10:00 Task Setup Building Task Directory
2023-04-04T03:14:24+10:00 Received Task received by client
```

relevant logs from nomad agent:
```
2023-04-03T17:14:25.885Z [DEBUG] client.alloc_runner.runner_hook: received result from CNI: alloc_id=2ff40152-c4b4-8ce8-9c7e-09ec416a9f88 result="{\"Interfaces\":{\"eth0\":{\"IPConfigs\":[{\"IP\":\"10.0.10.2\",\"Gateway\":\"10.0.10.1\"}],\"Mac\":\"3a:15:55:77:bc:cf\",\"Sandbox\":\"/var/run/docker/netns/4179007f7d56\"}},\"DNS\":[{}],\"Routes\":[{\"dst\":\"0.0.0.0/0\"},{\"dst\":\"169.254.1.1/32\",\"gw\":\"10.0.10.1\"}]}"
```

#### Expected Result
within nomad alloc status ${allocid}
You should see the container's ip:
something like:

```
Allocation Addresses:
Label Dynamic Address
80 no 10.0.10.2:80
```
yes im aware i have no port definition in the job but its a pointless thing to add given cni, however when adding a port definition)
```
+/- Job: "nginx"
+/- Task Group: "nginx" (1 create/destroy update)
+ Network {
Hostname: ""
+ MBits: "0"
+ Mode: "cni/frontend"
+ Dynamic Port {
+ HostNetwork: "default"
+ Label: "http"
+ To: "80"
}
}
- Network {
Hostname: ""
- MBits: "0"
- Mode: "cni/frontend"
}
Task: "nginx"

Scheduler dry-run:
- All tasks successfully allocated.
```

You see the allocation addresses look like this:
```
Allocation Addresses (mode = "cni/frontend"):
Label Dynamic Address
*http yes 172.25.1.70:24285 -> 80
```

#### Actual Result

Container ip no where to be found with inspection of the job/allocation

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the behavior with the shown CNI configuration and inspect the `nomad alloc status` and job status output paths. The change is complete when the allocation's CNI-issued IP is exposed in Nomad's CLI and UI, including the expected allocation address details.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, ubuntu
Domain
cli, frontend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.