hashicorp / hashicorp/nomad

Batch Job Priority

Open
#12,792 9 comments 6 reactions 0 assignees View on GitHub
hcc/jira stage/needs-discussion theme/batch theme/scheduling type/enhancement
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
Nomad v1.2.6 (a6c6b475db5073e33885377b4a5c733e1161020c)

### Operating system and Environment details

Linux ubuntu in Azure.

### Issue

With batch jobs, I would expect the allocations from higher priority job to be placed first.

In fact, I've just rewrote a bunch of our infra related to batch job submission only to find out this is not the case :man_facepalming: Is there any way to make sure the batch job allocations with higher priority are placed first?

We're running 100k+ task groups doing time series forecasting but I really need to be able to just run it and forget it and be sure that what goes in first comes out first, otherwise I run 10 jobs and instead of waiting 1 hours for the first to finish, I wait 10 hours for all of them to finish.

[Here](https://www.nomadproject.io/docs/internals/scheduling/scheduling) is a lot of talk about "priority ordering", so that probably means something else?

See also https://github.com/hashicorp/nomad/issues/9845

### Reproduction steps

Submit two batch jobs with different priorities.

#### Expected Result

Allocations from higher priority job are placed first.

#### Actual Result

Allocations are placed at random.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with Nomad's scheduling documentation linked in the issue and reproduce the behavior by submitting two batch jobs with different priorities. Trace how priority ordering is described for batch allocations and determine whether higher-priority allocations are expected to be placed first; done means the observed behavior matches a documented, tested priority rule.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.