Rebalance Nomad Scheduled Allocations

Nomad 1.0.3

Operating system and Environment details

PhotonOS3

Issue

Over time we have noticed our nodes getting allocations placed on the same node more and more (we use the spread algorithm as a default for the scheduler config). This has produced some interesting scenarios where we have system jobs that are unable to run due to dimensions being exhausted on the node BUT there are other nodes available for allocations on the saturated node.

Right now we manually restart the allocation group which results in the allocations starting to spread as expected but we are a little surprised by this behavior - and in a few instances an application and all it's instances are on the same node.
EDIT: We have a strong suspicion the above is the the culprit. We recently had a sizing operation done on some clusters and ALL of them that were affected by this sizing (and reboot) are displaying this inbalance. We will run tests to see if we can get verifiable behavior. More concerning is that applications seem to "stack up" like the screenshot below.

It may be coincidental but we have also recently migrated all these services to Service Mesh - Not sure that's relevant but I also don't want to exclude any significant changes that coincide with this observed behavior.

In addition we were wondering if there's a way we can issue a rebalance command across the cluster where the scheduler can move allocations? It would be really helpful in some scenarios we have where we might do a rolling restart through a cluster (of the actual nodes themselves) - what happens here is that the node when drains then starts the allocations on the remaining nodes as expected - but at the end of all this we end up with 1 node (the last one to restart) significantly unbalanced. In these scenarios we would really like to be able to trigger a rebalance to ensure the saturation described above does not happen.

Thanks!
Ian

Start by reproducing allocation placement after node reboots, drains, and manual allocation-group restarts, using the reported spread configuration. Review the scheduler's placement behavior and determine whether the imbalance is caused by reboot-related state or another change; done should include a verified cause and a defined approach for preventing or explicitly rebalancing allocations.