Azure / Azure/aksArc

[BUG] AksHci upgrade hangs when the upgrade is initiated after adding a physical node to the setup

Open
#288 0 comments 0 reactions 0 assignees View on GitHub
Bug
Dominant language
PowerShell
Stars
118
Forks
63
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**
AksHci upgrade hangs if we initiate the upgrade after adding a new physical to an existing AksHci setup. The hang typically looks as shown in the screenshot below.
![upgrade_hang](https://user-images.githubusercontent.com/176106/217862911-8aa2334a-1574-4e60-b43e-1d4e28afab58.png)

The CSI controller pod logs has the following error.
![csi_logs](https://user-images.githubusercontent.com/176106/217861842-d4ab3cfa-a423-4271-9856-24409f494082.png)

**To Reproduce**
Steps to reproduce the behavior:
1. Install AksHci
2. Add a new physical node
3. Perform AksHci upgrade with `Update-AksHci`. The upgrade hangs during this step

**Expected behavior**
Ideally, the upgrade should complete without any issues.

**Mitigation**
1. Drain the node using failover cluster UI as shown below.
![fc_tsg](https://user-images.githubusercontent.com/176106/217859970-b883b89a-cef2-40a2-b535-0677a50ef600.png)
Alternatively, you can use the command `Suspend-ClusterNode -Name -Drain` to drain the node.
2. Use `Remove-AksHciNode -nodeName ` to remove the machine from akshci setup
3. Use `Remove-ClusterNode -Name ` to remove the machine from failover cluster
4. Run `Update-AksHci` to trigger the upgrade.

Note: We can remove the node while the upgrade is hanging. The upgrade will proceed.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.