Restart mode change has no effect until job is stopped and rerun
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
Nomad v0.11.3
### Operating system and Environment details
macOS 10.15.5, Docker 19.03.8
### Issue
When a job's restart mode is updated from `delay` to `fail` the change does not take effect and existing allocs continue to restart using the old delay setting.
If the job is stopped and resubmitted the updated restart mode will take effect.
### Reproduction steps
1. Create `test.hcl`:
```
job "test" {
region = "global"
datacenters = ["dc1"]
type = "service"
group "service-group" {
count = 1
restart {
attempts = 1
interval = "10s"
delay = "1s"
mode = "delay"
}
reschedule {
delay = "5s"
delay_function = "constant"
unlimited = true
}
task "service" {
leader = true
driver = "docker"
config {
image = "alpine"
entrypoint = ["/bin/false"]
}
}
}
}
```
2. `nomad job run test.hcl`
3. Update `test.hcl` to `restart.mode = fail`.
4. `nomad job run test.hcl`
5. Existing alloc will continue to restart in `delay` mode.
This can also be reproduced by using `/bin/sleep infinity` as the command and killing the containers via docker, so it affects deployments to both healthy and unhealthy jobs.
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the behavior with test.hcl using the documented nomad job run steps, then trace how an updated restart.mode reaches existing allocations. Done means changing mode from delay to fail affects existing allocations without stopping and resubmitting the job, with regression coverage for both healthy and failing tasks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100