hashicorp / hashicorp/nomad

Nomad fails to launch jobs with empty CNI IPAM configuration

Open
#19,170 5 comments 0 reactions 0 assignees View on GitHub
stage/needs-discussion theme/cni theme/networking type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
Nomad v1.6.3
BuildDate 2023-10-30T12:58:10Z
Revision e0497bff14378d68cad76a801cc0eba93ce05039

nomad-podman-driver version 0.5.1

### Operating system and Environment details
Distributor ID: Ubuntu
Description: Ubuntu 22.04.3 LTS
Release: 22.04
Codename: jammy
podman version 3.4.4
macvlan CNI plugin version 0.4.0

### Issue

When using the macvlan CNI plugin with an empty IPAM configuration block running a job from nomad, the launch fails with the following error:

```
Setup Failure: failed to setup alloc: pre-run hook "network" failed: failed to configure networking for alloc: failed to configure network: no interface with an address
```

If a populated macvlan IPAM configuration specifying static, host-local, or dhcp is present, the container will launch with the intended IP address allocation on the macvlan container interface (e.g. eth0).

Empty IPAM configuration blocks (i.e. not containing static, host-local, or dhcp) in the macvlan CNI plugin should result in an interface-only environment - that is, a container should exist with a macvlan-paired interface but with no address assignment. This functions in non-nomad originated container deployments - for example, using podman directly to launch a container.

### Reproduction steps

macvlan CNI configuration with a populated host-local block:

1) CNI configuration

```
{
"cniVersion": "0.4.0",
"name": "macvlan-ssi",
"plugins": [
{
"type": "macvlan",
"mode": "private",
"master": "enp111s0f0",
"ipam": {
"type": "host-local",
"ranges": [
[
{
"subnet": "127.0.0.0/24",
"rangeStart": "127.0.0.128",
"rangeEnd": "127.0.0.160"
}
]
]
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
},
"snat": true
}
]
}
```

2) example job hcl - successful/working test case

```
job "bbtest2" {
datacenters = ["dc1"]

group "bbtest2" {
network {
mode = "cni/macvlan-ssi"
}

task "busybox" {
driver = "podman"

config {
image = "docker.io/busybox:latest"
entrypoint = [ "tail", "-f", "/dev/null" ]
cap_add = [ "net_admin" ]
}

resources {
cpu = 500
memory = 256
}
}
}
}
```

```
$ nomad job run busybox2.nomad.hcl
==> 2023-11-24T14:02:23Z: Monitoring evaluation "aa70c41a"
2023-11-24T14:02:23Z: Evaluation triggered by job "bbtest2"
2023-11-24T14:02:24Z: Evaluation within deployment: "cf4097fb"
2023-11-24T14:02:24Z: Allocation "1c68ced2" created: node "e56854cb", group "bbtest2"
2023-11-24T14:02:24Z: Evaluation status changed: "pending" -> "complete"
==> 2023-11-24T14:02:24Z: Evaluation "aa70c41a" finished with status "complete"
==> 2023-11-24T14:02:24Z: Monitoring deployment "cf4097fb"
✓ Deployment "cf4097fb" successful

2023-11-24T14:02:35Z
ID = cf4097fb
Job ID = bbtest2
Job Version = 6
Status = successful
Description = Deployment completed successfully

Deployed
Task Group Desired Placed Healthy Unhealthy Progress Deadline
bbtest2 1 1 1 0 2023-11-24T14:12:33Z

$ nomad job allocs bbtest2
ID Node ID Task Group Version Desired Status Created Modified
1c68ced2 e56854cb bbtest2 6 run running 19s ago 8s ago

$ nomad alloc exec 1c68 ip a
1: lo: mtu 65536 qdisc noop qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0@if4: mtu 1500 qdisc noqueue
link/ether 86:dc:4e:6b:f5:be brd ff:ff:ff:ff:ff:ff
inet 127.0.0.133/24 brd 127.0.0.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::84dc:4eff:fe6b:f5be/64 scope link
valid_lft forever preferred_lft forever
```

3) macvlan config with empty IPAM block (i.e. interface only, no address allocation)

```
{
"cniVersion": "0.4.0",
"name": "mvtest",
"plugins": [
{
"type": "macvlan",
"master": "enp111s0f0",
"ipam": {
}
}
]
}
```

The test network is intentionally kept simpler for purposes of reproduction only - a more populated configuration like the full one will result in the same error state.

4) modified example hcl - failure test case

```
job "bbtest3" {
datacenters = ["dc1"]

group "bbtest3" {
network {
mode = "cni/mvtest"
}

task "busybox" {
driver = "podman"

config {
image = "docker.io/busybox:latest"
entrypoint = [ "tail", "-f", "/dev/null" ]
cap_add = [ "net_admin" ]
}

resources {
cpu = 500
memory = 256
}
}
}
}
```

```
$ nomad job run busybox3.nomad.hcl
==> 2023-11-24T14:06:42Z: Monitoring evaluation "d938bf6c"
2023-11-24T14:06:42Z: Evaluation triggered by job "bbtest3"
2023-11-24T14:06:43Z: Evaluation within deployment: "83f2de85"
2023-11-24T14:06:43Z: Allocation "8cb91273" created: node "e56854cb", group "bbtest3"
2023-11-24T14:06:43Z: Evaluation status changed: "pending" -> "complete"
==> 2023-11-24T14:06:43Z: Evaluation "d938bf6c" finished with status "complete"
==> 2023-11-24T14:06:43Z: Monitoring deployment "83f2de85"
⠸ Deployment "83f2de85" in progress...

2023-11-24T14:06:43Z
ID = 83f2de85
Job ID = bbtest3
Job Version = 2
Status = running
Description = Deployment is running

Deployed
Task Group Desired Placed Healthy Unhealthy Progress Deadline
bbtest3 1 1 0 1 2023-11-24T14:16:42Z
```

Container remains in unhealthy state. From the nomad UI:

```
Setup Failure: failed to setup alloc: pre-run hook "network" failed: failed to configure networking for alloc: failed to configure network: no interface with an address
```

5) Manually launch the container using podman directly - successful test case

```
$ podman run -it --rm -d --cap-add=net_admin --network mvtest busybox:latest
c0a846007d898d181175462ab8d07c8a021a05b2d4fa28d103e89bc7791dd5ff

$ podman exec -it c0a8 ip a
1: lo: mtu 65536 qdisc noqueue qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0@if4: mtu 1500 qdisc noqueue
link/ether ce:67:8e:78:6a:3d brd ff:ff:ff:ff:ff:ff
inet6 fe80::cc67:8eff:fe78:6a3d/64 scope link
valid_lft forever preferred_lft forever
```

The container launches using the macvlan config with the empty IPAM block with an interface only, no IP address allocation.

Not reproduced here, but similar behavior is observed when using the docker driver rather than podman.

#### Expected Result

nomad launched containers should support empty CNI IPAM configuration blocks (i.e. "interface only" mode) in the same way the underlying container mechanism does.

#### Actual Result

nomad launched containers remain in Unhealthy state with the error specified in the reproduction steps.

### Job file (if appropriate)

Included in reproduction steps above.

### Nomad Client logs

```
2023-11-24T14:06:42.405Z [DEBUG] client.alloc_runner.runner_hook: received result from CNI: alloc_id=8cb91273-140b-b411-f872-956b5915dd35 result="{\"Interfaces\":{\"eth0\":{\"IPConfigs\":null,\"Mac\":\"02:86:40:92:91:2a\",\"Sandbox\":\"/var/run/netns/8cb91273-140b-b411-f872-956b5915dd35\"}},\"DNS\":[{}],\"Routes\":null}"
2023-11-24T14:06:42.405Z [ERROR] client.alloc_runner: prerun failed: alloc_id=8cb91273-140b-b411-f872-956b5915dd35 error="pre-run hook \"network\" failed: failed to configure networking for alloc: failed to configure network: no interface with an address"
2023-11-24T14:06:42.405Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=8cb91273-140b-b411-f872-956b5915dd35 task=busybox type="Setup Failure" msg="failed to setup alloc: pre-run hook \"network\" failed: failed to configure networking for alloc: failed to configure network: no interface with an address" failed=true
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the client.alloc_runner network pre-run hook and inspect how the CNI result is handled when Interfaces contains an interface with null IPConfigs. Reproduce the empty-IPAM job from the issue, compare it with the successful populated-IPAM case, and verify that an interface-only result completes without the “no interface with an address” failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.