Azure / Azure/AKS

[BUG] Windows node reports less CPU usage than its pods

Open
#4,028 39 comments 1 reaction 1 assignee Claimed by @allyford View on GitHub
action-required bug monitoring Needs Attention :wave: Under Investigation windows
Dominant language
TypeScript
Stars
2.1k
Forks
395
Avg merge
2d 22h
Merged PRs (30d)
13

Description

**Describe the bug**
`kubectl top node` command is showing less cpu load for windows nodes than expected.

**To Reproduce**
Steps to reproduce the behavior:
1. Create a nodepool with 1 Windows2022/2019 node running k8s 1.27.7 version (in my example: A2_v2, D8_v5 or B2ms)
2. Create simple cpu-consuming deployment. For example:
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: win-cpu-load
name: win-cpu-load
spec:
replicas: 3
selector:
matchLabels:
app: win-cpu-load
template:
metadata:
labels:
app: win-cpu-load
name: win-cpu-load
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.azure.com/agentpool
operator: In
values:
- test00
containers:
- name: windowswebserver
image: mcr.microsoft.com/windows/servercore:ltsc2022
command:
- powershell.exe
- -command
- "while ($true) { $result = 1; foreach ($number in 1..100000000) {$result = $result * $number}; Write-Host $result; Start-Sleep -Seconds 7; }"
```
4. Wait until metrics are collected (~1 minute)
5. Execute command `kubectl top pod` and `kubectl top node`
6. Pods have in total more cpu usage than node.

**Expected behavior**
Node should report more cpu usage than its pods in total

**Screenshots**
![image](https://github.com/Azure/AKS/assets/50938720/8886828b-f62d-4273-96db-dd996be1a9de)
(The pods from the left are assigned to the node from the right, three kube-system pods are not included, each generates aroung 100m)

**Environment (please complete the following information):**
- Node image version: AKSWindows-2022-containerd-20348.2113.231115 or AKSWindows-2019-containerd-17763.5122.231115
- Output of `kubectl version -o=yaml`
```yaml
clientVersion:
buildDate: "2023-09-13T09:35:49Z"
compiler: gc
gitCommit: 89a4ea3e1e4ddd7f7572286090359983e0387b2f
gitTreeState: clean
gitVersion: v1.28.2
goVersion: go1.20.8
major: "1"
minor: "28"
platform: windows/amd64
kustomizeVersion: v5.0.4-0.20230601165947-6ce0bf390ce3
serverVersion:
buildDate: "2023-10-19T00:14:21Z"
compiler: gc
gitCommit: 07a61d861519c45ef5c89bc22dda289328f29343
gitTreeState: clean
gitVersion: v1.27.7
goVersion: go1.20.10
major: "1"
minor: "27"
platform: linux/amd64
```

**Additional context**
The problem has appeared after upgrading aks cluster from 1.26.10 to 1.27.7.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.