CNI: provide `cni.Result` to task driver
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
Task drivers are not currently supplied much information about the network specification produced by CNI plugins. For example, consider the following CNI configuration:
```json
{
"name": "test",
"cniVersion": "0.4.0",
"plugins": [
{
"type": "bridge",
"name": "my-bridge",
"bridge": "virbr0",
"isDefaultGateway": true,
"ipMasq": true,
"hairpinMode": true,
"ipam": {
"type": "host-local",
"resolvConf": "/etc/resolv.conf",
"dataDir": "/run/networks",
"subnet": "192.168.40.0/24",
"rangeStart": "192.168.40.32",
"gateway": "192.168.40.1"
}
}
]
}
```
I've used this with a simple `raw_exec` task for purposes of illustration:
jobspec
```hcl
job "example" {
group "web" {
network {
mode = "cni/test"
}
task "http" {
driver = "raw_exec"
config {
command = "python3"
args = [
"-m", "http.server",
"--directory", "local",
]
}
}
}
}
```
The network manager reads the mode `cni/test` (ref [`network_manager_linux.go#L192-L197`](https://github.com/hashicorp/nomad/blob/v1.5.2/client/allocrunner/network_manager_linux.go#L192-L197) and calls `newCNINetworkConfigurator` in `networking_cni.go`, which ultimately calls [`Setup`](https://github.com/hashicorp/nomad/blob/v1.5.2/client/allocrunner/networking_cni.go#L88-L125). The debug log line we get here has plenty of detail about the resulting spec:
> 2023-03-23T14:37:04.978-0400 [DEBUG] client.alloc_runner.runner_hook: received result from CNI: alloc_id=078c18df-01cc-d337-6d82-4b0b5d9984d0 result="{\"Interfaces\":{\"eth0\":{\"IPConfigs\":[{\"IP\":\"192.168.40.36\",\"Gateway\":\"192.168.40.1\"}],\"Mac\":\"da:cc:07:05:a4:9d\",\"Sandbox\":\"/var/run/netns/078c18df-01cc-d337-6d82-4b0b5d9984d0\"},\"vethd60fa44f\":{\"IPConfigs\":null,\"Mac\":\"fe:26:66:49:d8:2d\",\"Sandbox\":\"\"},\"virbr0\":{\"IPConfigs\":null,\"Mac\":\"52:54:00:60:7f:fc\",\"Sandbox\":\"\"}},\"DNS\":[{}],\"Routes\":[{\"dst\":\"0.0.0.0/0\",\"gw\":\"192.168.40.1\"}]}"
But if we add a `spew.Dump` to the top of the task driver's [`StartTask`](https://github.com/hashicorp/nomad/blob/v1.5.2/drivers/rawexec/driver.go#L297), we get the following `NetworkIsolation` block:
```
NetworkIsolation: (*drivers.NetworkIsolationSpec)(0xc001a6cd20)({
Mode: (drivers.NetIsolationMode) (len=5) "group",
Path: (string) (len=51) "/var/run/netns/2ce20001-f442-73c0-60ab-2e74367fe444",
Labels: (map[string]string) {
},
HostsConfig: (*drivers.HostsConfig)()
}),
```
While this works fine for our built-in task drivers, @eveld has reported a need for custom task drivers to know about how the network was configured, so that they can set appropriate values in the task (for example, setting the IP in a Firecracker VM's [MMDS](https://github.com/firecracker-microvm/firecracker/blob/main/docs/mmds/mmds-user-guide.md)).
We currently pass _some_ of this information along for `bridge` networking if the driver has set some Docker-specific tags (ref [`network_hook.go#L132-L175`](https://github.com/hashicorp/nomad/blob/v1.5.2/client/allocrunner/network_hook.go#L132-L175), but this doesn't work for CNI and in any case shouldn't be Docker-specific.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the CNI flow in client/allocrunner/network_manager_linux.go and networking_cni.go, including Setup, then compare the existing data path in network_hook.go. Read the task-driver entry point at drivers/rawexec/driver.go StartTask and determine how the CNI result should reach custom drivers. Done means task drivers can access the resulting CNI network information without Docker-specific tags.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- networking
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100