A canary alloc with an unrecoverable error will stall a deployment indefinitely
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
Nomad v1.0.4+ent (72adb9f0e0ca0ac142f783fd62891840bc9029ba)`
### Operating system and Environment details
Ubuntu, internal hardened base image (can provide more details on the Slacks!)
### Issue
During a canary deploy, if a canary alloc lands on a node that fails to find docker auth, the deployment will become stalled forever and will not recover. That is, when the canary alloc fails to start due to missing docker auth:
* The alloc is not restarted (restarting would not be useful anyway if docker auth is broken)
* The alloc is not rescheduled (this might help if it were reschedule on a different node)
* The deployment does not auto revert even after hitting the healthy deadline or progress deadline, leading to a stalled job staying stalled until human operators intervene to manually `nomad deployment fail `
### Reproduction steps
Deploy a normal docker-driven Nomad job; it should use a public image, have a canary count of 1, auto revert true, auto promote true, and progress/healthy deadlines of ~ 5m.
Modify the job to pull a private image and then deploy it into a cluster that has no credentials or credential helpers capable of authing against the private repo.
#### Expected Result
Deployment will fail when one of the deadlines is met, and auto revert will roll things back to the previous stable state.
#### Actual Result
Canary allocation will fail hard with
```
2021-04-28T16:24:16-04:00 Alloc Unhealthy Unhealthy because of failed task
2021-04-28T16:24:16-04:00 Not Restarting Error was unrecoverable
2021-04-28T16:24:16-04:00 Driver Failure Failed to find docker auth for repo "": docker-credential-vault-login with input "" failed with stderr: exit status 1
```
The deployment does not auto revert, regardless of progress/healthy deadlines.
### Job file (if appropriate)
Happy to private a link to the HCP repo where it lives! We're an internal team so we can provide provide lots more detail if needed.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing Nomad's deployment handling for an unrecoverable canary allocation, especially the healthy and progress deadline paths and automatic revert behavior. Reproduce the failure with a canary deployment using a private Docker image and missing credentials; done means the deployment fails at a deadline and automatically reverts to the previous stable state.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go
- Domain
- distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100