hashicorp / hashicorp/nomad

progress_deadline distinguish between updates (auto_revert) and deployments

Open
#12,940 4 comments 1 reaction 0 assignees View on GitHub
type/enhancement
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Proposal
Add a new progress_deadline setting that only applies to updates and auto_revert decision making.

Today if you set progress_deadline to 5m and deploy a change to a job that, lets say, references a non-existing docker image, and set auto_revert to true; it will try for 5 minutes, fail (image not found), then revert to the old deployment.

This sounds good.

But what if, lets say the docker-registry is down for 15 minutes and you try to deploy a change?

I believe it will try for five minutes, fail, revert, continue to fail for five minutes (the problem wasnt the job, it was the docker-registry), and then give up...forever

Nomad giving up forever is a very bad and unexpected state. To get around it, maybe we want to set progress_deadline to 99999h (wish we could set unlimited = true here), but then you break the first case - auto_revert can never work.

So what I think I want is to tell nomad that if you are deciding whether to revert or not, consider the deadline 5m
But if you are deciding whether to fail the deployment, and wont revert, but will leave the job dead, consider a different deadline.

Why is the deadline for reverting the same as leaving the job dead forever?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the deployment behavior described in the issue for progress_deadline and auto_revert. Compare the non-existent image case with the registry outage case, then define separate deadline behavior that preserves auto-revert while avoiding a permanent failure state. Done means the semantics are agreed and covered by tests for both scenarios.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
devops, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.