hashicorp / hashicorp/nomad

Surprising behaviour with "Force New Periodic Instance" and prohibit_overlap

Open
#5,487 8 comments 2 reactions 0 assignees View on GitHub
theme/scheduling type/enhancement
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version

`Nomad v0.8.6 (ab54ebcfcde062e9482558b7c052702d4cb8aa1b+CHANGES)`

### Issue

We have some periodic jobs which will occasionally crash before completing. In these cases a developer may manually re-run the job to get a successful execution (taking care not to do so while another instance is running). However, we've noticed that the manually started job does not prevent Nomad from starting additional scheduled instances in the future. The resulting concurrent executions are problematic for some of our jobs.

The documentation for "[Force New Periodic Instance](https://www.nomadproject.io/api/jobs.html#force-new-periodic-instance)" does clearly specify that `prohibit_overlap` is ignored when creating the job instance. However, the fact that future instances will continue to be scheduled while the "forced" instance is still running was more surprising.

If this behavior is intended I think clarifying it in the documentation would be helpful.

And this should maybe be a separate feature request, but the ability to have `/v1/job/:job_id/periodic/force` respect `prohibit_overlap` would also be super helpful.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the Force New Periodic Instance section of the API documentation linked in the issue. Confirm how prohibit_overlap applies to forced and subsequently scheduled instances, then update the documentation to state the observed behavior and its limits. Treat honoring prohibit_overlap in the force endpoint as a separate feature request.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.