Placement failures after job update despite eligible clients and available resources
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
Output from `nomad version`
Nomad v1.11.3
BuildDate 2026-03-11T09:10:27Z
Revision 173ab08a0210789da531847c4ce3c3518f7fb34b
### Operating system and Environment details
23~24.04.1-Ubuntu
### Issue
We are seeing Nomad placement failures even though there are available clients that match the job constraints and appear to have sufficient free resources.
This can happen after a job update. Nomad stops the current allocation, but then fails to place the replacement allocation, including cases where it should be able to place it back onto the same client.
What makes this look incorrect:
there are healthy clients respecting the job constraints those clients appear to have enough available resources for the allocation restarting the Nomad service on one of the eligible clients allows the placement to happen
This makes it look like Nomad may be holding stale client or scheduling state after the update, or otherwise incorrectly considering eligible clients as unavailable until the client agent is restarted.
### Reproduction steps
We have not found a deterministic reproduction yet, but the issue has been observed with the following pattern:
- Run a job on a client that satisfies the job constraints and has sufficient available resources.
- Update the job in a way that causes the existing allocation to stop and a replacement allocation to be scheduled.
- Observe that the replacement allocation sometimes enters placement failure instead of being placed on an eligible client, including the same client that was previously running the allocation.
- Verify that there are still healthy clients matching the constraints with enough free resources.
- Restart the Nomad service on one of the eligible clients.
- Observe that placement often succeeds after the client restart, though not in every occurrence.
Because this does not happen consistently, it may require repeating the update flow multiple times before the failure appears.
#### Expected Result
If a job update stops an allocation and there are eligible clients with sufficient resources, Nomad should place the new allocation without requiring a Nomad client restart.
#### Actual Result
After the update, the allocation enters placement failure even though eligible clients are available. Restarting Nomad on an available client allows the allocation to be placed.
#### Workaround
Restart the Nomad service on eligible clients.
### Job file (if appropriate)
### Nomad Server logs (if appropriate)
### Nomad Client logs (if appropriate)
on the available client I am seeing a lot of
2026-05-22T22:28:51.951Z [INFO] agent: (runner) creating watcher
2026-05-22T22:28:51.952Z [INFO] agent: (runner) starting
2026-05-22T22:35:41.531Z [INFO] agent: (runner) stopping
2026-05-22T22:35:41.531Z [INFO] agent: (runner) received finish
2026-05-22T22:35:41.537Z [INFO] agent: (runner) stopping
2026-05-22T22:35:41.537Z [INFO] agent: (runner) received finish
2026-05-22T22:35:43.697Z [INFO] agent: (runner) stopping
2026-05-22T22:35:43.697Z [INFO] agent: (runner) received finish
2026-05-22T22:35:43.703Z [INFO] agent: (runner) stopping
2026-05-22T22:35:43.704Z [INFO] agent: (runner) received finish
2026-05-22T22:35:43.802Z [INFO] agent: (runner) stopping
2026-05-22T22:35:43.802Z [INFO] agent: (runner) received finish
2026-05-22T22:35:44.055Z [INFO] agent: (runner) stopping
2026-05-22T22:35:44.056Z [INFO] agent: (runner) received finish
2026-05-22T22:35:51.627Z [INFO] agent: (runner) stopping
2026-05-22T22:35:51.628Z [INFO] agent: (runner) received finish
2026-05-22T22:36:04.193Z [INFO] agent: (runner) stopping
2026-05-22T22:36:04.193Z [INFO] agent: (runner) received finish
2026-05-22T22:36:04.253Z [INFO] agent: (runner) creating new runner (dry: false, once: false)
2026-05-22T22:36:04.253Z [INFO] agent: (runner) creating watcher
2026-05-22T22:36:04.253Z [INFO] agent: (runner) starting
2026-05-22T22:36:09.608Z [INFO] agent: (runner) stopping
2026-05-22T22:36:09.608Z [INFO] agent: (runner) received finish
2026-05-22T22:38:08.168Z [INFO] agent: (runner) creating new runner (dry: false, once: false)
2026-05-22T22:38:08.168Z [INFO] agent: (runner) creating watcher
2026-05-22T22:38:08.169Z [INFO] agent: (runner) starting
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the reported job-update flow and reviewing the Nomad client logs around placement failure, allocation stopping, and client restart. Compare scheduler behavior before and after restarting an eligible client; done means identifying a reproducible cause and confirming that eligible replacement allocations place without a client restart.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100