hashicorp / hashicorp/nomad

A canary alloc with an unrecoverable error will stall a deployment indefinitely

Open
#10,465 2 comments 0 reactions 0 assignees View on GitHub
stage/needs-investigation theme/deployments theme/restart/reschedule theme/scheduling type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
Nomad v1.0.4+ent (72adb9f0e0ca0ac142f783fd62891840bc9029ba)`

### Operating system and Environment details
Ubuntu, internal hardened base image (can provide more details on the Slacks!)

### Issue

During a canary deploy, if a canary alloc lands on a node that fails to find docker auth, the deployment will become stalled forever and will not recover. That is, when the canary alloc fails to start due to missing docker auth:

* The alloc is not restarted (restarting would not be useful anyway if docker auth is broken)
* The alloc is not rescheduled (this might help if it were reschedule on a different node)
* The deployment does not auto revert even after hitting the healthy deadline or progress deadline, leading to a stalled job staying stalled until human operators intervene to manually `nomad deployment fail `

### Reproduction steps

Deploy a normal docker-driven Nomad job; it should use a public image, have a canary count of 1, auto revert true, auto promote true, and progress/healthy deadlines of ~ 5m.

Modify the job to pull a private image and then deploy it into a cluster that has no credentials or credential helpers capable of authing against the private repo.

#### Expected Result

Deployment will fail when one of the deadlines is met, and auto revert will roll things back to the previous stable state.

#### Actual Result

Canary allocation will fail hard with

```
2021-04-28T16:24:16-04:00 Alloc Unhealthy Unhealthy because of failed task
2021-04-28T16:24:16-04:00 Not Restarting Error was unrecoverable
2021-04-28T16:24:16-04:00 Driver Failure Failed to find docker auth for repo "": docker-credential-vault-login with input "" failed with stderr: exit status 1
```

The deployment does not auto revert, regardless of progress/healthy deadlines.

### Job file (if appropriate)

Happy to private a link to the HCP repo where it lives! We're an internal team so we can provide provide lots more detail if needed.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing Nomad's deployment handling for an unrecoverable canary allocation, especially the healthy and progress deadline paths and automatic revert behavior. Reproduce the failure with a canary deployment using a private Docker image and missing credentials; done means the deployment fails at a deadline and automatically reverts to the previous stable state.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
distributed-systems, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.