Batch Job Priority
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
Nomad v1.2.6 (a6c6b475db5073e33885377b4a5c733e1161020c)
### Operating system and Environment details
Linux ubuntu in Azure.
### Issue
With batch jobs, I would expect the allocations from higher priority job to be placed first.
In fact, I've just rewrote a bunch of our infra related to batch job submission only to find out this is not the case :man_facepalming: Is there any way to make sure the batch job allocations with higher priority are placed first?
We're running 100k+ task groups doing time series forecasting but I really need to be able to just run it and forget it and be sure that what goes in first comes out first, otherwise I run 10 jobs and instead of waiting 1 hours for the first to finish, I wait 10 hours for all of them to finish.
[Here](https://www.nomadproject.io/docs/internals/scheduling/scheduling) is a lot of talk about "priority ordering", so that probably means something else?
See also https://github.com/hashicorp/nomad/issues/9845
### Reproduction steps
Submit two batch jobs with different priorities.
#### Expected Result
Allocations from higher priority job are placed first.
#### Actual Result
Allocations are placed at random.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with Nomad's scheduling documentation linked in the issue and reproduce the behavior by submitting two batch jobs with different priorities. Trace how priority ordering is described for batch allocations and determine whether higher-priority allocations are expected to be placed first; done means the observed behavior matches a documented, tested priority rule.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100