containers / containers/nri-plugins

topology-aware: cpu limits are not considered in scoring when no pool remains with enough shared pool capacity

Open
#670 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
114
Forks
44
Avg merge
1d 8h
Merged PRs (30d)
20

Description

### Bug Description

When no candidate pool can satisfy a Burstable container's CPU limit, the scoring logic treats all insufficient pools as tied and can prefer a much smaller lower-level topology pool over a much larger higher-level pool. This can make the effective cpuset far smaller than the CPU limit, without CFS throttling visibility.

The means that the pod will silently be subject to a much smaller burst-able range.

### Expected behavior

Pools are sorted on the remaining capacity. The pool with the largest capacity remaining is chosen to maximize the cpus consumable through cpu.max.

### Experienced behavior

Applied the same request=6, limits=80 pod to a node with 128 cores 6 times. Exec into the node to find the following cpuset configuration, where the last pod got a pool that is much smaller:
```
=== kub-3118-burst80-1 ===
cpuset.cpus=1-55,57-127
cpu.max=8000000 100000
=== kub-3118-burst80-2 ===
cpuset.cpus=1-55,57-127
cpu.max=8000000 100000
=== kub-3118-burst80-3 ===
cpuset.cpus=1-55,57-127
cpu.max=8000000 100000
=== kub-3118-burst80-4 ===
cpuset.cpus=1-55,57-127
cpu.max=8000000 100000
=== kub-3118-burst80-5 ===
cpuset.cpus=1-55,57-127
cpu.max=8000000 100000
=== kub-3118-burst80-6 ===
cpuset.cpus=44-47,108-111
cpu.max=8000000 100000
```
```
I: [ nri-plugin ] => CreateContainer default/kub-3118-burst80-1/main (65e13d44d7dbf70bdc3e4e5f8286b581fee807f3286a20345332a6bd48365635)
I: [ policy ] * applying grant
I: [ policy ] => pinning default/kub-3118-burst80-1/main to (shared) cpuset 1-55,57-127
I: [ nri-plugin ] => CreateContainer default/kub-3118-burst80-2/main (ca32aed8b90de2b58768fdbfea258a386b8add1829ac84ad8125fe060fc2939f)
I: [ policy ] * applying grant
I: [ policy ] => pinning default/kub-3118-burst80-2/main to (shared) cpuset 1-55,57-127
I: [ nri-plugin ] => CreateContainer default/kub-3118-burst80-3/main (590c640e6529257521b59af7db6d323c051f9120a656e3cdb7f39873ff1e8e37)
I: [ policy ] * applying grant
I: [ policy ] => pinning default/kub-3118-burst80-3/main to (shared) cpuset 1-55,57-127
I: [ nri-plugin ] => CreateContainer default/kub-3118-burst80-4/main (42c6bca6a999c2360186a6315928e0be301eb429075c7d2cc7d18e9f941ad7d6)
I: [ policy ] * applying grant
I: [ policy ] => pinning default/kub-3118-burst80-4/main to (shared) cpuset 1-55,57-127
I: [ nri-plugin ] => CreateContainer default/kub-3118-burst80-5/main (cb9f2b2318bf703ef567873f7959ae1c33055830a0544ca7ad388b5c59a1387a)
I: [ policy ] * applying grant
I: [ policy ] => pinning default/kub-3118-burst80-5/main to (shared) cpuset 1-55,57-127
I: [ nri-plugin ] => CreateContainer default/kub-3118-burst80-6/main (149bea792ebbda985413a7da486927996aef12ab1b166b84864308558ea4df3e)
I: [ policy ] * applying grant
I: [ policy ] => pinning default/kub-3118-burst80-6/main to (shared) cpuset 44-47,108-111
```

### Steps to reproduce the behavior

Steps to reproduce the behavior
1. Apply the pod. Done on a node with 128 cores, so scoring will place it at the node level pool due to high cpu limits.
```
# Apply
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: high-cpu-burst-test
labels:
test: high-cpu-burst
spec:
restartPolicy: Never
containers:
- name: main
image: busybox:1.36
command: ["sleep", "3600"]
resources:
requests:
cpu: "6"
memory: "256Mi"
limits:
cpu: "80"
memory: "512Mi"
EOF

# Wait for Ready
kubectl wait --for=condition=Ready pod/high-cpu-burst-test --timeout=180s

# Inspect
kubectl get pod high-cpu-burst-test -o wide
kubectl describe pod high-cpu-burst-test
```
2. Repeat many times until no pool remains that has 80 cores. Apply one last time
3. Examine the cpuset for each pod. The last pod should get a leaf node of the resource tree.

### Environment

- nri-plugins version: 0.12.2
- nri plugin(s) running: topology aware
- container runtime: 1.7xx
- OS:
- kernel:
- Kubernetes version: 1.34

### Additional context

I understand that since the pods are burstable, the behavior is not technically incorrect there were are no guarantees for satisfying cpu.max anyways. However, it seems that in these cases, we should prefer at least pools which more remaining capacity (ie. socket level compared to L3 level).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.