Batch jobs not getting preempted with cores
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
Client: 1.4.3
Server: 1.4.4
### Operating system and Environment details
```
root@ns5020866:/home/ubuntu# nomad operator scheduler get-config
Scheduler Algorithm = spread
Memory Oversubscription = true
Reject Job Registration = false
Pause Eval Broker = false
Preemption System Scheduler = true
Preemption Service Scheduler = true
Preemption Batch Scheduler = true
Preemption SysBatch Scheduler = true
Modify Index = 5
```
### Issue
Batch jobs with lower priority don't get preempted in favor of batch jobs with higher priority.
### Reproduction steps
Example of two jobs (assuming a single node running with 12 cores)
`job1.nomad`
```
job "job1" {
datacenters = ["dc1"]
type = "batch"
group "group1" {
restart {
attempts = 0
mode = "fail"
}
task "task1" {
driver = "exec"
resources {
cores = 11
memory = 300
}
config {
command = "/bin/sleep"
args = ["200"]
}
}
}
}
```
`job2.nomad`
```
job "job2" {
datacenters = ["dc1"]
type = "batch"
priority = 100
group "group2" {
restart {
attempts = 0
mode = "fail"
}
task "task2" {
driver = "exec"
resources {
cores = 11
memory = 400
}
config {
command = "/bin/sleep"
args = ["200"]
}
}
}
}
```
```
nomad job run job1.nomad
nomad job run job2.nomad
```
#### Expected Result
job1 should have been preempted and evicted in favour of job2 as `priority(job1) = 50 < priority(job2) = 100`
#### Actual Result
job2 shows resources exhausted
```
root@ns1018760:/home/ubuntu# nomad job run d.nomad
==> 2023-02-20T14:25:05Z: Monitoring evaluation "cc0e8a92"
2023-02-20T14:25:06Z: Evaluation triggered by job "test9"
2023-02-20T14:25:06Z: Evaluation status changed: "pending" -> "complete"
==> 2023-02-20T14:25:06Z: Evaluation "cc0e8a92" finished with status "complete" but failed to place all allocations:
2023-02-20T14:25:06Z: Task Group "test9" (failed to place 1 allocation):
* Resources exhausted on 1 nodes
* Dimension "cores" exhausted on 1 nodes
2023-02-20T14:25:06Z: Evaluation "98bd3281" waiting for additional capacity to place remainder
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the scheduler behavior with the two provided batch job definitions on a single 12-core node, using the shown scheduler configuration and checking the placement output. Done means a higher-priority batch job preempts the lower-priority job instead of reporting cores exhausted.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100