Deployment lasted beyond progress deadline
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
Output from `nomad version`
Nomad v1.1.1 (7feec97c04de4f8afff54ca9e56d66a61dfbfeb3)
### Operating system and Environment details
Amazon Linux 2
### Issue
We experienced a job deployment that lasted much longer than the progress deadline. Each taskgroup's progress deadline was 10 minutes, but this deployment was running for 7 days!
This on-going deployment lead to a series of side-effects, including effectively disabling autoscaling actions from the Nomad autoscaler.

### Reproduction steps
Unfortunately, we are unsure how to reproduce this, but the deployment was triggered by the Nomad autoscaler. We have been running v1.1.1 /w the Nomad autoscaler for a few months and this is the first time we've seen this.
#### Expected Result
Deployment to fail after progress deadline expires.
#### Actual Result
Deployment continued on until it was found by one of our engineers during an investigation to high load on our production service.
Contributor guide
No contributing guide indexed for this repository
Research direction
No files or tests are named. Start by tracing how Nomad handles deployment progress deadlines for deployments triggered by the Nomad autoscaler, using the reported seven-day deployment as the behavior to investigate. Done means a deployment fails when its task groups exceed the ten-minute progress deadline and does not remain active indefinitely.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100