hashicorp / hashicorp/nomad

Nomad schedules more instances of system task per node after node reboot

Open
#9,846 7 comments 0 reactions 0 assignees View on GitHub
stage/needs-investigation theme/system-scheduler type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
Nomad v0.12.0 (8f7fbc8e7b5a4ed0d0209968faf41b238e6d5817)

### Operating system and Environment details
CentOs 7.5 VM on Virtualbox 6.1, 3 nodes cluster, server and client on same node

### Issue
Nomad schedules more instances of system task per node after node reboot

### Reproduction steps
- Config the nomad client with `client {gc_max_allocs = 1}`
- Starts the job with job file below, `nomad job run `
- Check the status of the job are running
```
nomad job status test
ID = test
Name = test
Submit Date = 2021-01-18T23:29:56Z
Type = system
Priority = 50
Datacenters = dc1
Namespace = default
Status = running
Periodic = false
Parameterized = false

Summary
Task Group Queued Starting Running Failed Complete Lost
test 0 0 3 0 0 0

Allocations
ID Node ID Task Group Version Desired Status Created Modified
0a861162 1d358dc0 test 0 run running 26m24s ago 24m45s ago
3989bac4 93eba225 test 0 run running 26m24s ago 24m48s ago
d5f19cff 8af71708 test 0 run running 26m24s ago 24m44s ago
```

- reboot all the 3 nodes, at the same time
- Check the status, if its still 3 active tasks(pending or running) status, like below, do the reboot 3 nodes again
```
nomad job status test
ID = test
Name = test
Submit Date = 2021-01-18T23:29:56Z
Type = system
Priority = 50
Datacenters = dc1
Namespace = default
Status = running
Periodic = false
Parameterized = false

Summary
Task Group Queued Starting Running Failed Complete Lost
test 0 1 2 0 2 0

Allocations
ID Node ID Task Group Version Desired Status Created Modified
3cd842f1 1d358dc0 test 0 run running 4m54s ago 4m42s ago
0a861162 1d358dc0 test 0 run complete 32m54s ago 4m52s ago
3989bac4 93eba225 test 0 run running 32m54s ago 4m36s ago
d5f19cff 8af71708 test 0 run pending 32m54s ago 43s ago
```

- At last, it will launch more than one task on one node, like
```
nomad job status test
ID = test
Name = test
Submit Date = 2021-01-18T23:29:56Z
Type = system
Priority = 50
Datacenters = dc1
Namespace = default
Status = running
Periodic = false
Parameterized = false

Summary
Task Group Queued Starting Running Failed Complete Lost
test 0 2 3 0 5 0

Allocations
ID Node ID Task Group Version Desired Status Created Modified
05d00f19 8af71708 test 0 run running 27m26s ago 27m26s ago
ae08bf8e 1d358dc0 test 0 run pending 27m26s ago 27m26s ago
91b19a4b 93eba225 test 0 run running 30m10s ago 27m10s ago
3cd842f1 1d358dc0 test 0 run pending 36m26s ago 8s ago
0a861162 1d358dc0 test 0 run complete 1h4m ago 27m22s ago
3989bac4 93eba225 test 0 run complete 1h4m ago 27m22s ago
d5f19cff 8af71708 test 0 run running 1h4m ago 27m26s ago
```

### Job file (if appropriate)

```
job "test" {
datacenters = ["dc1"]
type = "system"

group "test" {
restart {
interval = "6m"
attempts = 10
delay = "10s"
mode = "delay"
}

# add prestart task
task "test-pre" {
driver = "docker"
lifecycle {
hook = "prestart"
sidecar = false
}

config {
image = "alpine:3.8"
command = "sh"

args = ["-c", "echo test > /alloc/test_file"]
}
}

task "test" {
driver = "docker"

config {
image = "alpine:3.8"
command = "sh"

args = ["-c", "if [ ! -s /alloc/test_file ]; then sleep 5; exit 1; else while sleep 3600; do :; done; fi"]
}
}
}
}
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.