[BUG] Cluster Autoscaler clashes with Scale-down Mode deallocate
- Dominant language
- TypeScript
- Stars
- 2.1k
- Forks
- 395
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 13
Description
**Describe the bug**
When using scale-down mode deallocate in combination with the cluster autoscaler, the Not-Ready nodes in the cluster can cause autoscaler to stop considering node pools or to stop working completely.
When using scale-down mode deallocate, on scale-down nodes are only deallocated, leaving them in the cluster with the state Not Ready.
Cluster Autoscaler on the other hand has the settings `ok-total-unready-count` and `max-total-unready-percentage` with defaults values of 3 and 45% respectively. Once the number of deallocated nodes in a node pool reaches 3, the node pool will no longer be scaled and once the complete cluster reaches 45% Not Ready nodes, the autoscaler will halt completely.
**To Reproduce**
Use multiple nodepools within a cluster and enable scale-down mode deallocate. First scale to like 10 nodes and afterwards scale-down to 1 node so that 9 nodes get deallocated. Afterwards schedule workload on the cluster. The nodepool will be disregarded, and autoscaler will e.g. log:
`Found: 9 instances in deallocated state, returning target size: 0 for scaleSet scaleSetName`
**Expected behavior**
In general, I would wish that autoscaler and scale-down mode Deallocate would work together, but there are multiple solutions to this issue:
* inform the user that when using scale-down mode deallocate that he needs to address those settings. Easiest solution, but more or less disables the auto-scaler settings which exist for a reason
* no longer mark deallocated nodes as Not Ready but remove them from the cluster.
* make auto-scaler aware that there is a difference between failed nodes and deallocate nodes, and he should only stop if the nodes are failed (i.e. ignore deallocated nodes for this limit)
**Environment (please complete the following information):**
- Kubernetes version: 1.31.10
Contributor guide
Assessment
This issue has not been assessed yet.