hashicorp / hashicorp/nomad

Batch jobs not getting preempted with cores

Open
#16,232 1 comment 0 reactions 0 assignees View on GitHub
stage/accepted theme/preemption theme/scheduling type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
Client: 1.4.3
Server: 1.4.4

### Operating system and Environment details
```
root@ns5020866:/home/ubuntu# nomad operator scheduler get-config
Scheduler Algorithm = spread
Memory Oversubscription = true
Reject Job Registration = false
Pause Eval Broker = false
Preemption System Scheduler = true
Preemption Service Scheduler = true
Preemption Batch Scheduler = true
Preemption SysBatch Scheduler = true
Modify Index = 5
```

### Issue

Batch jobs with lower priority don't get preempted in favor of batch jobs with higher priority.

### Reproduction steps

Example of two jobs (assuming a single node running with 12 cores)

`job1.nomad`
```
job "job1" {
datacenters = ["dc1"]
type = "batch"

group "group1" {
restart {
attempts = 0
mode = "fail"
}
task "task1" {
driver = "exec"
resources {
cores = 11
memory = 300
}
config {
command = "/bin/sleep"
args = ["200"]
}
}

}
}
```

`job2.nomad`
```
job "job2" {
datacenters = ["dc1"]
type = "batch"
priority = 100

group "group2" {
restart {
attempts = 0
mode = "fail"
}

task "task2" {
driver = "exec"
resources {
cores = 11
memory = 400
}
config {
command = "/bin/sleep"
args = ["200"]
}
}

}
}
```

```
nomad job run job1.nomad
nomad job run job2.nomad
```

#### Expected Result
job1 should have been preempted and evicted in favour of job2 as `priority(job1) = 50 < priority(job2) = 100`

#### Actual Result
job2 shows resources exhausted
```
root@ns1018760:/home/ubuntu# nomad job run d.nomad
==> 2023-02-20T14:25:05Z: Monitoring evaluation "cc0e8a92"
2023-02-20T14:25:06Z: Evaluation triggered by job "test9"
2023-02-20T14:25:06Z: Evaluation status changed: "pending" -> "complete"
==> 2023-02-20T14:25:06Z: Evaluation "cc0e8a92" finished with status "complete" but failed to place all allocations:
2023-02-20T14:25:06Z: Task Group "test9" (failed to place 1 allocation):
* Resources exhausted on 1 nodes
* Dimension "cores" exhausted on 1 nodes
2023-02-20T14:25:06Z: Evaluation "98bd3281" waiting for additional capacity to place remainder
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the scheduler behavior with the two provided batch job definitions on a single 12-core node, using the shown scheduler configuration and checking the placement output. Done means a higher-priority batch job preempts the lower-priority job instead of reporting cores exhausted.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.