hashicorp / hashicorp/nomad

CNI: provide `cni.Result` to task driver

Open
#16,624 3 comments 3 reactions 0 assignees View on GitHub
hcc/jira theme/cni theme/networking type/enhancement
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

Task drivers are not currently supplied much information about the network specification produced by CNI plugins. For example, consider the following CNI configuration:

```json
{
"name": "test",
"cniVersion": "0.4.0",
"plugins": [
{
"type": "bridge",
"name": "my-bridge",
"bridge": "virbr0",
"isDefaultGateway": true,
"ipMasq": true,
"hairpinMode": true,
"ipam": {
"type": "host-local",
"resolvConf": "/etc/resolv.conf",
"dataDir": "/run/networks",
"subnet": "192.168.40.0/24",
"rangeStart": "192.168.40.32",
"gateway": "192.168.40.1"
}
}
]
}
```

I've used this with a simple `raw_exec` task for purposes of illustration:

jobspec

```hcl
job "example" {
group "web" {

network {
mode = "cni/test"
}

task "http" {

driver = "raw_exec"
config {
command = "python3"
args = [
"-m", "http.server",
"--directory", "local",
]
}
}
}
}
```

The network manager reads the mode `cni/test` (ref [`network_manager_linux.go#L192-L197`](https://github.com/hashicorp/nomad/blob/v1.5.2/client/allocrunner/network_manager_linux.go#L192-L197) and calls `newCNINetworkConfigurator` in `networking_cni.go`, which ultimately calls [`Setup`](https://github.com/hashicorp/nomad/blob/v1.5.2/client/allocrunner/networking_cni.go#L88-L125). The debug log line we get here has plenty of detail about the resulting spec:

> 2023-03-23T14:37:04.978-0400 [DEBUG] client.alloc_runner.runner_hook: received result from CNI: alloc_id=078c18df-01cc-d337-6d82-4b0b5d9984d0 result="{\"Interfaces\":{\"eth0\":{\"IPConfigs\":[{\"IP\":\"192.168.40.36\",\"Gateway\":\"192.168.40.1\"}],\"Mac\":\"da:cc:07:05:a4:9d\",\"Sandbox\":\"/var/run/netns/078c18df-01cc-d337-6d82-4b0b5d9984d0\"},\"vethd60fa44f\":{\"IPConfigs\":null,\"Mac\":\"fe:26:66:49:d8:2d\",\"Sandbox\":\"\"},\"virbr0\":{\"IPConfigs\":null,\"Mac\":\"52:54:00:60:7f:fc\",\"Sandbox\":\"\"}},\"DNS\":[{}],\"Routes\":[{\"dst\":\"0.0.0.0/0\",\"gw\":\"192.168.40.1\"}]}"

But if we add a `spew.Dump` to the top of the task driver's [`StartTask`](https://github.com/hashicorp/nomad/blob/v1.5.2/drivers/rawexec/driver.go#L297), we get the following `NetworkIsolation` block:

```
NetworkIsolation: (*drivers.NetworkIsolationSpec)(0xc001a6cd20)({
Mode: (drivers.NetIsolationMode) (len=5) "group",
Path: (string) (len=51) "/var/run/netns/2ce20001-f442-73c0-60ab-2e74367fe444",
Labels: (map[string]string) {
},
HostsConfig: (*drivers.HostsConfig)()
}),
```

While this works fine for our built-in task drivers, @eveld has reported a need for custom task drivers to know about how the network was configured, so that they can set appropriate values in the task (for example, setting the IP in a Firecracker VM's [MMDS](https://github.com/firecracker-microvm/firecracker/blob/main/docs/mmds/mmds-user-guide.md)).

We currently pass _some_ of this information along for `bridge` networking if the driver has set some Docker-specific tags (ref [`network_hook.go#L132-L175`](https://github.com/hashicorp/nomad/blob/v1.5.2/client/allocrunner/network_hook.go#L132-L175), but this doesn't work for CNI and in any case shouldn't be Docker-specific.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the CNI flow in client/allocrunner/network_manager_linux.go and networking_cni.go, including Setup, then compare the existing data path in network_hook.go. Read the task-driver entry point at drivers/rawexec/driver.go StartTask and determine how the CNI result should reach custom drivers. Done means task drivers can access the resulting CNI network information without Docker-specific tags.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
networking
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.