Skipping Update Stanza when task group is renamed
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
Nomad seems to only honor the job's `update` stanza when task group details change, but not when the task group layout structure changes. For example, if a task group is renamed, Nomad will immediately shutdown all current allocations and start new ones and potentially cause a minor outage.
The reason is that generic scheduler (and reconciler) considers the updates for each task group independently. If a job renames a task group, the scheduler will treat the change as a task group removal and insertion.
We should consider documenting this explicitly if it's not documented already. Also, we can consider having max_parallel apply globally to all task groups, so we would only shutdown one allocation at a time
### Reproduction steps
Run the following commands
```sh
# run the first job, command reports 3 allocations
nomad job run ./canary-test.nomad
# update env-var value - see a single canary job starting
cat ./canary-test.nomad | sed 's|"val1"|"val2"|g' | nomad job run -
# check that there is a single running deployment with a single canary allocation
nomad job status example
# now submit the job but with group name change, note
cat ./canary-test.nomad | sed 's|"cache-0"|"cache-1"|g' | nomad job run -
nomad job status example
```
My sample output
```sh
$ nomad job run ./canary-test.nomad
==> Monitoring evaluation "0cf9bedf"
Evaluation triggered by job "example"
Allocation "67f44ee8" created: node "624b8edb", group "cache-0"
Allocation "d3764a91" created: node "624b8edb", group "cache-0"
Allocation "60110964" created: node "624b8edb", group "cache-0"
Evaluation within deployment: "03b744f3"
Allocation "d3764a91" status changed: "pending" -> "running" (Tasks are running)
Allocation "60110964" status changed: "pending" -> "running" (Tasks are running)
Allocation "67f44ee8" status changed: "pending" -> "running" (Tasks are running)
Evaluation status changed: "pending" -> "complete"
==> Evaluation "0cf9bedf" finished with status "complete"
$ # update env-var value - see canary
$ cat ./canary-test.nomad | sed 's|"val1"|"val2"|g' | nomad job run -
==> Monitoring evaluation "26a19a5e"
Evaluation triggered by job "example"
Evaluation within deployment: "2a5cf58c"
Allocation "0f98a673" created: node "624b8edb", group "cache-0"
Evaluation status changed: "pending" -> "complete"
==> Evaluation "26a19a5e" finished with status "complete"
$ # notice deployment has one canary
$ nomad job status example
ID = example
Name = example
Submit Date = 2020-08-09T20:14:57-04:00
Type = service
Priority = 50
Datacenters = dc1
Namespace = default
Status = running
Periodic = false
Parameterized = false
Summary
Task Group Queued Starting Running Failed Complete Lost
cache-0 0 0 4 0 0 0
Latest Deployment
ID = 2a5cf58c
Status = running
Description = Deployment is running but requires manual promotion
Deployed
Task Group Promoted Desired Canaries Placed Healthy Unhealthy Progress Deadline
cache-0 false 3 1 1 0 0 2020-08-09T20:24:57-04:00
Allocations
ID Node ID Task Group Version Desired Status Created Modified
0f98a673 624b8edb cache-0 1 run running 8s ago 7s ago
60110964 624b8edb cache-0 0 run running 1m8s ago 57s ago
67f44ee8 624b8edb cache-0 0 run running 1m8s ago 57s ago
d3764a91 624b8edb cache-0 0 run running 1m8s ago 57s ago
$ # now submit the job but with group name change
$ cat ./canary-test.nomad | sed 's|"cache-0"|"cache-1"|g' | nomad job run -
==> Monitoring evaluation "24838f8b"
Evaluation triggered by job "example"
Allocation "f2ad41f7" created: node "624b8edb", group "cache-1"
Allocation "7165ad7e" created: node "624b8edb", group "cache-1"
Allocation "9c7e5681" created: node "624b8edb", group "cache-1"
Evaluation within deployment: "9b4d6fef"
Evaluation status changed: "pending" -> "complete"
==> Evaluation "24838f8b" finished with status "complete"
$ nomad job status example
ID = example
Name = example
Submit Date = 2020-08-09T20:15:49-04:00
Type = service
Priority = 50
Datacenters = dc1
Namespace = default
Status = running
Periodic = false
Parameterized = false
Summary
Task Group Queued Starting Running Failed Complete Lost
cache-0 0 0 0 0 4 0
cache-1 0 0 3 0 0 0
Latest Deployment
ID = 9b4d6fef
Status = running
Description = Deployment is running
Deployed
Task Group Desired Placed Healthy Unhealthy Progress Deadline
cache-1 3 3 0 0 2020-08-09T20:25:49-04:00
Allocations
ID Node ID Task Group Version Desired Status Created Modified
f2ad41f7 624b8edb cache-1 2 run running 7s ago 6s ago
7165ad7e 624b8edb cache-1 2 run running 7s ago 6s ago
9c7e5681 624b8edb cache-1 2 run running 7s ago 6s ago
0f98a673 624b8edb cache-0 1 stop complete 59s ago 6s ago
60110964 624b8edb cache-0 0 stop complete 1m59s ago 6s ago
67f44ee8 624b8edb cache-0 0 stop complete 1m59s ago 6s ago
d3764a91 624b8edb cache-0 0 stop complete 1m59s ago 6s ago
```
### Job file (if appropriate)
```hcl
job "example" {
datacenters = ["dc1"]
update {
max_parallel = 1
health_check = "task_states"
auto_promote = false
canary = 1
stagger = "30s"
}
group "cache-0" {
count = 3
task "redis" {
driver = "docker"
config {
image = "redis:3.2"
}
env {
ENV_KEY = "val1"
}
resources {
cpu = 500
memory = 256
}
}
}
}
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the generic scheduler and reconciler behavior described in the issue, then reproduce the task-group rename using the provided Nomad commands and job file. Determine how the update stanza and max_parallel are expected to apply when a group is removed and inserted. Done should include an agreed behavior, with documentation or implementation and tests covering the rename scenario.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100