Nomad schedules more instances of system task per node after node reboot
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
Nomad v0.12.0 (8f7fbc8e7b5a4ed0d0209968faf41b238e6d5817)
### Operating system and Environment details
CentOs 7.5 VM on Virtualbox 6.1, 3 nodes cluster, server and client on same node
### Issue
Nomad schedules more instances of system task per node after node reboot
### Reproduction steps
- Config the nomad client with `client {gc_max_allocs = 1}`
- Starts the job with job file below, `nomad job run `
- Check the status of the job are running
```
nomad job status test
ID = test
Name = test
Submit Date = 2021-01-18T23:29:56Z
Type = system
Priority = 50
Datacenters = dc1
Namespace = default
Status = running
Periodic = false
Parameterized = false
Summary
Task Group Queued Starting Running Failed Complete Lost
test 0 0 3 0 0 0
Allocations
ID Node ID Task Group Version Desired Status Created Modified
0a861162 1d358dc0 test 0 run running 26m24s ago 24m45s ago
3989bac4 93eba225 test 0 run running 26m24s ago 24m48s ago
d5f19cff 8af71708 test 0 run running 26m24s ago 24m44s ago
```
- reboot all the 3 nodes, at the same time
- Check the status, if its still 3 active tasks(pending or running) status, like below, do the reboot 3 nodes again
```
nomad job status test
ID = test
Name = test
Submit Date = 2021-01-18T23:29:56Z
Type = system
Priority = 50
Datacenters = dc1
Namespace = default
Status = running
Periodic = false
Parameterized = false
Summary
Task Group Queued Starting Running Failed Complete Lost
test 0 1 2 0 2 0
Allocations
ID Node ID Task Group Version Desired Status Created Modified
3cd842f1 1d358dc0 test 0 run running 4m54s ago 4m42s ago
0a861162 1d358dc0 test 0 run complete 32m54s ago 4m52s ago
3989bac4 93eba225 test 0 run running 32m54s ago 4m36s ago
d5f19cff 8af71708 test 0 run pending 32m54s ago 43s ago
```
- At last, it will launch more than one task on one node, like
```
nomad job status test
ID = test
Name = test
Submit Date = 2021-01-18T23:29:56Z
Type = system
Priority = 50
Datacenters = dc1
Namespace = default
Status = running
Periodic = false
Parameterized = false
Summary
Task Group Queued Starting Running Failed Complete Lost
test 0 2 3 0 5 0
Allocations
ID Node ID Task Group Version Desired Status Created Modified
05d00f19 8af71708 test 0 run running 27m26s ago 27m26s ago
ae08bf8e 1d358dc0 test 0 run pending 27m26s ago 27m26s ago
91b19a4b 93eba225 test 0 run running 30m10s ago 27m10s ago
3cd842f1 1d358dc0 test 0 run pending 36m26s ago 8s ago
0a861162 1d358dc0 test 0 run complete 1h4m ago 27m22s ago
3989bac4 93eba225 test 0 run complete 1h4m ago 27m22s ago
d5f19cff 8af71708 test 0 run running 1h4m ago 27m26s ago
```
### Job file (if appropriate)
```
job "test" {
datacenters = ["dc1"]
type = "system"
group "test" {
restart {
interval = "6m"
attempts = 10
delay = "10s"
mode = "delay"
}
# add prestart task
task "test-pre" {
driver = "docker"
lifecycle {
hook = "prestart"
sidecar = false
}
config {
image = "alpine:3.8"
command = "sh"
args = ["-c", "echo test > /alloc/test_file"]
}
}
task "test" {
driver = "docker"
config {
image = "alpine:3.8"
command = "sh"
args = ["-c", "if [ ! -s /alloc/test_file ]; then sleep 5; exit 1; else while sleep 3600; do :; done; fi"]
}
}
}
}
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.