Azure / Azure/azure-container-networking

CNI Windows Add DelegatedNIC - scan for any left-over HNS Networks before trying to find master interface

Open
#4,540 4 comments 0 reactions 0 assignees View on GitHub
cns windows
Dominant language
Go
Stars
434
Forks
276
Avg merge
5d 20h
Merged PRs (30d)
33

Description

We saw an issue where a Windows CNI add call was killed mid-process (right after it invoked the `CreateNetwork` HNS call, but before that call returned to CNI), so HNS proceeded (as it was commanded) to create an HNS network for a delegated NIC

*Before* that `CreateNetwork` call finished in HNS, CNI also then got a DELETE call for the same pod (since containerd automatically invoked the CNI delete after the CNI add failed)

The CNI DELETE call basically did nothing, since CNI had not yet written anything to state since the ADD call was killed mid-process

After the CNI DELETE call finished, the background HNS createNetwork call *completed*, so then there was a "leaked" HNS Network (HNS Network exists on the VM, but CNI has no idea about it and did not save it to any state)

Sequence of events:
1. CNI ADD call starts
2. CNI issues `CreateNetwork` request https://github.com/Azure/azure-container-networking/blob/20d09ab789046047e6cecb4cd1a40e99af936c2f/network/network_windows.go#L362
3. HNS begins creating the network for the delegated nic (1-1 mapping)
4. CNI ADD call is killed! The last log line we see is "`Creating hcn network`", nothing after that (not `ADD command completed with error`, nothing)
5. CNI DELETE call starts
6. CNI does **not** issue delete on HNS Network (because it was not able to save it to any state)
7. CNI DELETE call ends (basically did nothing)
8. HNS createNetwork call succeeds in the background (but CNI knows nothing about this network, because CNI was killed before it could save the network to any state)
9. There is now a drifting HNS Network on the VM, holding the MAC up
10. New CNI ADD call comes in for the same MAC
11. CNI cannot find the MAC, because it is now tied up to the old, leaked HNS network

## One natural thought
Would be to say "Hey, let's store the HNS network that we are about to create, so that in case CNI add call gets killed mid-HNS network create, the CNI Delete will clean it up"
- Wrong, HNS does not have deterministic behavior if we issue an HNS network delete, on a HNS Network that is *in progress* of being created. It is then a race, as to which call finishes (could be still that HNS network delete call finishes first, and then the HNS network create call finishes last, so we still have a leaked HNS network)

## Better solution
1. During a CNI Add (for windows, delegated nic), first, do an audit of all the HNS Networks on the VM
2. See if any HNS Networks don't belong to any pods (in which case, they have been leaked)
3. Delete the network
4. Then proceed with CNI ADD call as normal
5. https://github.com/Azure/azure-container-networking/blob/20d09ab789046047e6cecb4cd1a40e99af936c2f/cni/network/network.go#L731

Contributor guide

Open the contributing guide

Research direction

Start at cni/network/network.go around line 731 and network/network_windows.go around line 362, then trace the Windows delegated-NIC ADD flow and its HNS CreateNetwork call. Determine how to identify HNS networks that belong to no pod and remove them before the normal ADD proceeds; done means leaked networks are cleaned up without introducing a delete/create race.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.