hashicorp / hashicorp/nomad

Nomad is unable to reschedule pending jobs.

Open
#18,645 9 comments 1 reaction 0 assignees View on GitHub
stage/needs-investigation stage/waiting-reply theme/scheduling type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version

Output from `nomad version`

```
Nomad v1.6.2
BuildDate 2023-09-13T16:47:25Z
Revision 73e372ad94033db2ceaf53468b270a31544c23fd

```

### Operating system and Environment details

AmazonLinux 2023, 2nd generation Intel Xeon Scalable Processors (Cascade Lake 8275CL)

### Issue

Nomad isn’t able to schedule `pending` jobs when more nodes become available. For example:
When 1 machine is alive (Machine A), and nomad schedules 40 jobs out of which machine A has capacity for 25, there are 15 pending jobs. Autoscaler adds Machine B, but none of the 15 pending jobs are scheduled even when there are more resources available.

### Reproduction steps

If we send 200 jobs to nomad to schedule at once, we can see that it struggled to re-schedule the pending jobs once the autoscaler adds more machines.

### Expected Result

When a new machine comes up, the pending jobs are scheduled.

### Actual Result

When a new machine comes up, no pending jobs are not scheduled and still keep being pending.

### Example eval of a job:

Example eval [here](https://gist.github.com/gs0510/3d081938c5a266c52bd7a0cd2bf830f3).

We also saw this on the alloc status,
```
Placement Failure
Task Group "run-cell":
* Class "worker": 3 nodes excluded by filter
* Constraint "${node.class} = box": 3 nodes excluded by filter
* Resources exhausted on 5 nodes
* Class "box" exhausted on 5 nodes
* Dimension "cores" exhausted on 3 nodes
* Dimension "memory" exhausted on 2 nodes
```

The above is false because the machine barely had any jobs running, and definitely did not run out of either cores or memory.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the scheduling failure with pending jobs and a newly available machine, then inspect the placement-failure evaluation shown in the issue. Trace why the scheduler does not reconsider pending jobs and reports exhausted resources incorrectly; done means pending jobs are placed after capacity appears and the placement diagnostics reflect actual resources.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.