Nomad is unable to reschedule pending jobs.
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
Output from `nomad version`
```
Nomad v1.6.2
BuildDate 2023-09-13T16:47:25Z
Revision 73e372ad94033db2ceaf53468b270a31544c23fd
```
### Operating system and Environment details
AmazonLinux 2023, 2nd generation Intel Xeon Scalable Processors (Cascade Lake 8275CL)
### Issue
Nomad isn’t able to schedule `pending` jobs when more nodes become available. For example:
When 1 machine is alive (Machine A), and nomad schedules 40 jobs out of which machine A has capacity for 25, there are 15 pending jobs. Autoscaler adds Machine B, but none of the 15 pending jobs are scheduled even when there are more resources available.
### Reproduction steps
If we send 200 jobs to nomad to schedule at once, we can see that it struggled to re-schedule the pending jobs once the autoscaler adds more machines.
### Expected Result
When a new machine comes up, the pending jobs are scheduled.
### Actual Result
When a new machine comes up, no pending jobs are not scheduled and still keep being pending.
### Example eval of a job:
Example eval [here](https://gist.github.com/gs0510/3d081938c5a266c52bd7a0cd2bf830f3).
We also saw this on the alloc status,
```
Placement Failure
Task Group "run-cell":
* Class "worker": 3 nodes excluded by filter
* Constraint "${node.class} = box": 3 nodes excluded by filter
* Resources exhausted on 5 nodes
* Class "box" exhausted on 5 nodes
* Dimension "cores" exhausted on 3 nodes
* Dimension "memory" exhausted on 2 nodes
```
The above is false because the machine barely had any jobs running, and definitely did not run out of either cores or memory.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the scheduling failure with pending jobs and a newly available machine, then inspect the placement-failure evaluation shown in the issue. Trace why the scheduler does not reconsider pending jobs and reports exhausted resources incorrectly; done means pending jobs are placed after capacity appears and the placement diagnostics reflect actual resources.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100