hashicorp / hashicorp/nomad

Restart mode change has no effect until job is stopped and rerun

Open
#8,350 1 comment 0 reactions 0 assignees View on GitHub
stage/accepted type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
Nomad v0.11.3

### Operating system and Environment details
macOS 10.15.5, Docker 19.03.8

### Issue
When a job's restart mode is updated from `delay` to `fail` the change does not take effect and existing allocs continue to restart using the old delay setting.

If the job is stopped and resubmitted the updated restart mode will take effect.

### Reproduction steps
1. Create `test.hcl`:
```
job "test" {
region = "global"
datacenters = ["dc1"]

type = "service"

group "service-group" {
count = 1

restart {
attempts = 1
interval = "10s"
delay = "1s"
mode = "delay"
}

reschedule {
delay = "5s"
delay_function = "constant"
unlimited = true
}

task "service" {
leader = true

driver = "docker"

config {
image = "alpine"
entrypoint = ["/bin/false"]
}
}
}
}
```
2. `nomad job run test.hcl`
3. Update `test.hcl` to `restart.mode = fail`.
4. `nomad job run test.hcl`
5. Existing alloc will continue to restart in `delay` mode.

This can also be reproduced by using `/bin/sleep infinity` as the command and killing the containers via docker, so it affects deployments to both healthy and unhealthy jobs.

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the behavior with test.hcl using the documented nomad job run steps, then trace how an updated restart.mode reaches existing allocations. Done means changing mode from delay to fail affects existing allocations without stopping and resubmitting the job, with regression coverage for both healthy and failing tasks.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.