Competing Azure CNI installation modes in e2e tests
- Dominant language
- Go
- Stars
- 159
- Forks
- 277
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 178
Description
While debugging flaky e2e tests I've noticed that `/var/log/azure-vnet.log` has logs which are obviously from two different `azure-vnet` binaries.
For example, I ran:
```
KEEP_VMSS=1 go test -v -timeout 90m -run '^Test_AzureLinuxV2_AzureCNI$'
```
Here are the first 10 lines from `azure-vnet.log`:
```
2025/11/27 15:09:12 [5672] [cni] reboot time 2025-11-27 15:08:22 +0000 UTC
2025/11/27 15:09:12 [5672] Connection to telemetry socket failed: dial unix /var/run/azure-vnet-telemetry.sock: connect: no such file or directory
2025/11/27 15:09:12 [5672] [Azure-Utils] pkill -f azure-vnet-telemetry
2025/11/27 15:09:12 [5672] [Telemetry] Starting telemetry service process :/opt/cni/bin/azure-vnet-telemetry args:[-d /opt/cni/bin]
2025/11/27 15:09:12 [5672] [Telemetry] Telemetry service started
2025/11/27 15:09:12 [5692] [cni] reboot time 2025-11-27 15:08:22 +0000 UTC
2025/11/27 15:09:12 [5706] [cni] reboot time 2025-11-27 15:08:22 +0000 UTC
2025/11/27 15:09:12 [5672] Connected to telemetry service
2025/11/27 15:09:12 [5672] [cni-net] Plugin azure-vnet version v1.1.8.
2025/11/27 15:09:12 [5672] [cni-net] Running on Linux version 5.15.186.1-1.cm2 (root@CBL-Mariner) (gcc (GCC) 11.2.0, GNU ld (GNU Binutils) 2.37) #1 SMP Tue Jul 29 02:54:43 UTC 2025
```
And here are more lines from further down the log file:
```
{"level":"info","ts":"2025-11-27T15:09:27.399Z","caller":"./main.go:93","msg":"Environment variable set","pid":7191,"component":"cni-main","CNI_COMMAND":"ADD"}
{"level":"info","ts":"2025-11-27T15:09:27.401Z","caller":"store/json.go:196","msg":"Acquiring process lock","pid":7191,"component":"cni-store"}
{"level":"info","ts":"2025-11-27T15:09:27.401Z","caller":"store/json.go:215","msg":"Acquired process lock with timeout value of","pid":7191,"component":"cni-store","timeout":30}
{"level":"info","ts":"2025-11-27T15:09:27.402Z","caller":"telemetry/telemetrybuffer.go:383","msg":"Connected to telemetry service","pid":7191,"component":"cni-main"}
{"level":"info","ts":"2025-11-27T15:09:27.402Z","caller":"network/network.go:166","msg":"Plugin Info","pid":7191,"component":"cni-net","name":"azure-vnet","version":"v1.6.32"}
```
As can be seen, the **format** of the top lines is unstructured text whereas the bottom is JSON. In addition, we can see that the top lines are from `v1.1.8` whereas the bottom is `v1.6.32`.
Turns out that nowadays we use a DaemonSet named `azure-cns` with an init container named `cni-installer` to deploy the Azure CNI to cluster nodes. Therefore, when the user specifies `--network-plugin=azure` when creating a cluster, RP passes `none` to AB here:
https://github.com/Azure/AgentBaker/blob/d992c05685f28f78a5126fdfad93c54c0df97695/parts/linux/cloud-init/artifacts/cse_install.sh#L140
As a result, the CSE script doesn't install the plugin.
**However**, looks like in the e2e tests we still specify `azure` as the network plugin in tests which use the Azure CNI:
https://github.com/Azure/AgentBaker/blob/79d2559bb88c03dbf9a4a99f5f966ca9bfb311a9/e2e/scenario_test.go#L102-L114
As a result, we now have **two competing mechanisms** for installing the CNI plugin in the e2e tests. I suspect that this has led to all sorts of race conditions and weird behavior.
Contributor guide
Research direction
Start with e2e/scenario_test.go around lines 102-114 and compare its Azure CNI network-plugin configuration with the cse_install.sh behavior linked in the issue. Run KEEP_VMSS=1 go test -v -timeout 90m -run '^Test_AzureLinuxV2_AzureCNI$' and inspect /var/log/azure-vnet.log. Done means the e2e test uses one CNI installation mechanism and no longer produces mixed azure-vnet versions or formats.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, go, kubernetes
- Domain
- networking, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100