Azure / Azure/AKS

[BUG] Deallocated nodes are being reported as "Unhealthy"

Open
#4,313 57 comments 5 reactions 1 assignee Assigned to @kevinkrp93 View on GitHub
bug Needs Attention :wave: nodepools Under Investigation
Dominant language
TypeScript
Stars
2.1k
Forks
395
Avg merge
2d 22h
Merged PRs (30d)
13

Description

**Describe the bug**
When node pool `--scale-down-mode` is being set to Deallocate and scale in 1 node after then, it will cause Unhealty report on Azure portal.

**To Reproduce**
```bash
# Basic parameter for set-up
aks=
rG=
location=southeastasia

# Create resource group
az group create -n ${rG} -l ${location}

# Set `--node-os-upgrade-channel` to None in case cannot scale/change nodepool properties later
az aks create -n ${aks} -g ${rG} \
--nodepool-name agentpool \
--node-count 2 \
--node-os-upgrade-channel None \
--no-ssh-key

# Set to Deallocate for reproducing
az aks nodepool update --name agentpool --cluster-name ${aks} -g ${rG} \
--scale-down-mode Deallocate

# Reproducing issue by scaling in one node
az aks nodepool scale --name agentpool --cluster-name ${aks} -g ${rG} \
--node-count 1

# Giving time for AKS to fully processing
sleep 120

# Checking node ready status
# Issue reproduced here
az aks get-credentials -n ${aks} -g ${rG}
kubectl get node
```

```bash
# Try if stop/start fix the issue
# Stop AKS
az aks stop -n ${aks} -g ${rG}
# Start AKS
az aks start -n ${aks} -g ${rG}

# Issue persists
kubectl get node
```
(Health warning popped up like after ~30 mins after restarting the AKS (part 2 of the script), simply wait and check Azure portal in 30 mins. Make sure the page is being fully refreshed by using `Ctrl+F5`.)

**Expected behavior**
The deallocated nodes should be removed from AKS node list.

**Screenshots**
![image](https://github.com/Azure/AKS/assets/142381267/5fad885f-a3d3-4eb0-ba6b-c1712ef38f37)
![image](https://github.com/Azure/AKS/assets/142381267/e8ff0060-20e2-429b-9b95-a4fa3e8b2123)
![image](https://github.com/Azure/AKS/assets/142381267/d95c128a-0a4c-4857-8dae-0c5fb540840c)
"The node could not be found in running/powered on state."
![image](https://github.com/Azure/AKS/assets/142381267/5b657c2b-7f08-4b81-a555-e5909a534c75)
![image](https://github.com/Azure/AKS/assets/142381267/50524771-b336-4e88-944e-f8b8943ec008)
![image](https://github.com/Azure/AKS/assets/142381267/1f5adf12-ebf6-4117-9448-b96b0710a494)
![image](https://github.com/Azure/AKS/assets/142381267/756e86ef-f765-499b-bff3-2f8c326a77fc)
(Why there are still 5 Pods on supposed-not-started node? I don't know.)
![image](https://github.com/Azure/AKS/assets/142381267/f17c3228-0118-4977-9c93-a7ab508cdf4f)

**Environment (please complete the following information):**
- CLI Version: 2.61.0
- Kubernetes version: 1.28.9
- CLI Extension version: N/A
- Browser: Cloud Shell

**Additional context**
In my memory, this issue persists a long time ago, I just start to dig the root cause today.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.