containers / containers/nri-plugins

topology-aware: internal error from changing containers' NUMA nodes by adjusting AvailableResources

Open
#92 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
114
Forks
44
Avg merge
1d 8h
Merged PRs (30d)
20

Description

Assume that a container runs on CPUs of NUMA node 0.

An admin wants to reorganize server resources so that containers will not use CPUs on NUMA/die/socket 0 anymore by removing those CPUs from AvailableResources.

When this is done, restarting the topology aware NRI plugin with new configuration fails with an internal error:

```
E0710 07:30:57.289447 1 nri.go:784] <= Synchronize FAILED: failed to start policy topology-aware: topology-aware: failed to start:
topology-aware: failed to restore allocations from cache:
topology-aware: failed to allocate from :
topology-aware: internal error: NUMA node #1: can't slice 3 exclusive CPUs from , 0m available
```

Let's discuss if this is a bug, expected behavior or if we should provide a configuration option for forcing new CPU/memory pinning, even if it would lead into costly memory accesses/moves.

Current workaround on this error is deleting the cache and thereby forcing reassignment of resources from scratch. Using this workaround or draining a node before AvailableResources change are both heavier operations than what forcing new pinning would be.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.