When canaries fail in a deployment with lesser count than its predecessor, the culled allocations are never restored
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
I've encountered a situation (In Nomad 1.7.3.dev) where Nomad eagerly culls previous-version allocations when a new job version deployment calls for a lower count, even when there are canaries involved, and does not reinstate the culled allocations in the case that the new deployment fails.
Per https://github.com/hashicorp/nomad/issues/5033#issuecomment-451045074 (and internal discussion with the team), this seems like a bug.
### Reproduction steps
- Run a job (with 20 allocs, and 10 canaries), everything places and runs successfully. v0.
- Update the job, modifying:
- something about the tasks that cause them to fail.
- the task group `count` down from 20 to 15
- This deployment (v1) results in a deployment failing and we fall back to v0 as the live state of affairs. My v0 running allocations had not been shut down by the deployment, because it waited to see if they would succeed, and they didn’t. Back to v0.
- Except.... 5 of them did shut down. Before the v1 deployment ever failed, it looks like Nomad eagerly shuts down “extra” v0 allocs whose numbers were greater than the new count.
#### Expected Result
- v0 of my job's allocs has 20 running allocations (the v0 count)
#### Actual Result
- v0 of my job's allocs has 15 running allocations (the v1 and failed deployment count)
### Job file (if appropriate)
```
job "fails_every_10" {
update {
healthy_deadline = "60s"
progress_deadline = "1h"
auto_revert = false
canary = 10
max_parallel = 10
}
datacenters = ["dc1", "dc2"]
constraint {
attribute = "${attr.kernel.name}"
operator = "set_contains_any"
value = "darwin,linux"
}
type = "service"
group "grouper" {
count = 20
task "roll dice" {
driver = "raw_exec"
resources {
cpu = 50
memory = 64
}
config {
command = "node"
args = ["-e", <
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.