[Feedback] auto-repair network after memory-preserving live migration of VMSS instances using peered VNETs
- Dominant language
- TypeScript
- Stars
- 2.1k
- Forks
- 395
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 13
Description
**Describe your scenario**
Azure is increasingly using live-migration after hardware failures. Which are happening under the hood for AKS. Most of the time these issues can go unnoticed but we have noticed for subset of the live migrations where the network is restored few hours later.
| **Network Type** | **Status After VM Freeze** | **Fixed After Kubelet Restart?** |
|------------------------------------------|----------------------------|----------------------------------|
| **Within Same VNET** | ✅ Works | |
| **Pod-to-Pod (Same Node)** | ✅ Works | |
| **Pod-to-Pod (Cross Node, Same VNET, Azure CNI)** | ✅ Works | |
| **Pod-to-Pod (Cross Node, Peered VNETs)** | ❌ Broken | ✅ Yes |
| **Pod-to-External (Internet)** | ✅ Works | |
| **Pod-to-Node (Peered VNETs)** | ❌ Broken | ✅ Yes |
I have been trying to investigate to provide some compelling evidence on what exactly triggers the auto-healing after few hours and so far the most sensible fix I have been able to found is a kubelet restart. Which recovers the network connectivity across peered VNETs.
below are some of the observed conditions on the affected node
### **SR-IOV VF Reset**
```log
Dec 19 19:56:30 hv_netvsc eth0: VF registering: eth1
Dec 19 19:56:30 mlx5_core 8748:00:02.0 eth1: joined to eth0
Dec 19 19:56:30 hv_netvsc eth0: Data path switched to VF: eth1
```
### Traffic Control (TC) and Connection Tracking (CT)
```
Dec 19 19:56:30 mlx5_core 8748:00:02.0: mlx5_tc_ct_init: tc ct offload not supported
```
### walinux agent errors
```
Dec 19 19:56:30 python3[2426]: ERROR ExtHandler Error fetching goal state: [ProtocolError] Error fetching goal state: [ResourceGoneError]
```
**Feedback**
This issue causes interruptions in multi-region setups, particularly those that heavily rely on peered VNETs for cross-region communication.
One potential solution is to drain and discard the affected node, a topic already under discussion in
https://github.com/Azure/AKS/issues/3463 which would be great but I believe if AKS network team could take a look at the impact of VM Freeze events on peered VNETs it might not to be too complex to repair the node network.
Contributor guide
Assessment
This issue has not been assessed yet.