hashicorp / hashicorp/nomad

Deployment lasted beyond progress deadline

Open
#11,267 1 comment 2 reactions 0 assignees View on GitHub
stage/needs-investigation theme/deployments type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
Output from `nomad version`
Nomad v1.1.1 (7feec97c04de4f8afff54ca9e56d66a61dfbfeb3)

### Operating system and Environment details
Amazon Linux 2

### Issue
We experienced a job deployment that lasted much longer than the progress deadline. Each taskgroup's progress deadline was 10 minutes, but this deployment was running for 7 days!

This on-going deployment lead to a series of side-effects, including effectively disabling autoscaling actions from the Nomad autoscaler.

Screen Shot 2021-10-05 at 3 14 36 PM
Screen Shot 2021-10-05 at 3 25 10 PM

### Reproduction steps
Unfortunately, we are unsure how to reproduce this, but the deployment was triggered by the Nomad autoscaler. We have been running v1.1.1 /w the Nomad autoscaler for a few months and this is the first time we've seen this.

#### Expected Result
Deployment to fail after progress deadline expires.

#### Actual Result
Deployment continued on until it was found by one of our engineers during an investigation to high load on our production service.

Contributor guide

No contributing guide indexed for this repository

Research direction

No files or tests are named. Start by tracing how Nomad handles deployment progress deadlines for deployments triggered by the Nomad autoscaler, using the reported seven-day deployment as the behavior to investigate. Done means a deployment fails when its task groups exceed the ten-minute progress deadline and does not remain active indefinitely.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.