TaskGroup ends in Queued & Complete state after client node lost
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
Nomad v0.11.4 (3c6abec8d4a5c1af96ac3e1910a90949390dea19)
### Operating system and Environment details
CentOS 7.8.2003 on bare-metal
### Issue
I have a batch job with 42 individual tasks wrapped in 42 groups. During run, one of the client nodes failed and left the Nomad cluster altogether. If I run `nomad job status` on this job, the 2 groups that landed on that failed node show a summary state of one "Queued" and one "Complete". Like so,
```
ID = gmx-covid19_complex_tmprss2-5505-ps1_2020-08-25-0
Name = gmx-covid19_complex_tmprss2-5505-ps1_2020-08-25-0
Submit Date = 2020-08-25T20:31:09-07:00
Type = batch
Priority = 50
Datacenters = hq
Namespace = default
Status = running
Periodic = false
Parameterized = false
Summary
Task Group Queued Starting Running Failed Complete Lost
grp_1 0 0 1 0 0 0
...
grp_13 1 0 0 0 1 0 <= NOTE: Queued & Complete
...
grp_17 1 0 0 0 1 0 <= NOTE: Queued & Complete
...
Allocations
ID Node ID Task Group Version Desired Status Created Modified
792be1a8 399a6eac grp_1 0 run running 10h35m ago 10h35m ago
...
9e194fb5 0566cec4 grp_13 0 stop complete 2d15h ago 2d6m ago <= NOTE: stop & complete
...
1d6be2ca 0566cec4 grp_17 0 stop complete 2d18h ago 2d6m ago <= NOTE: stop & complete
```
(`grp_1` is included as an example of normal.)
Since the failed node is no longer part of the Nomad cluster, `nomad alloc restart ` failed with a 500 error stating the node doesn't exist. If instead I issue a `nomad alloc stop ` on these 2 allocations, no error occurs, but the "Queued" states changed to "0".
From what I can see, the tasks are lost and it does not appear Nomad will reallocate them elsewhere.
So here are 2 issues:
* Why are the groups identified as "Queued"? It does not appear they will be reallocated.
* It would be very useful to be able to force Nomad to reallocate these lost groups.
### Reproduction steps
I'm afraid you'd have to take out a client node while the allocations are running to reproduce this.
### Job file (if appropriate)
This is the trimmed output of `curl http://127.0.0.1:4646/v1/job/myJob`:
```
{
"Stable" : false,
"ID" : "myJob",
"CreateIndex" : 226771,
"Affinities" : null,
"Status" : "running",
"JobModifyIndex" : 226771,
"TaskGroups" : [
{
"Affinities" : null,
"ReschedulePolicy" : {
"Interval" : 86400000000000,
"Attempts" : 1,
"Unlimited" : false,
"MaxDelay" : 0,
"DelayFunction" : "constant",
"Delay" : 5000000000
},
"ShutdownDelay" : null,
"Meta" : null,
"RestartPolicy" : {
"Interval" : 86400000000000,
"Attempts" : 25,
"Mode" : "fail",
"Delay" : 1800000000000
},
"Networks" : null,
"Services" : null,
"Update" : null,
"Tasks" : [
{
"Affinities" : null,
"Templates" : null,
"CSIPluginConfig" : null,
"ShutdownDelay" : 0,
"User" : "",
"Config" : {
"volumes" : [
"/mnt/shared/myJob/input:/run.d/input:ro",
"/mnt/shared/myJob/output:/run.d/output",
],
"args" : [
"--verbose",
"--uid=1000043",
"--gid=100"
],
"image" : "myJob/image:1",
"network_mode" : "none"
},
"Leader" : false,
"Meta" : null,
"RestartPolicy" : {
"Interval" : 86400000000000,
"Attempts" : 25,
"Mode" : "fail",
"Delay" : 1800000000000
},
"Resources" : {
"Networks" : null,
"Devices" : null,
"IOPS" : 0,
"DiskMB" : 0,
"CPU" : 20000,
"MemoryMB" : 4000
},
"Services" : null,
"Vault" : null,
"KillTimeout" : 20000000000,
"Artifacts" : null,
"Constraints" : null,
"Lifecycle" : null,
"Env" : {
"SYSTEM" : "complex",
"GROUP" : "group1"
},
"Kind" : "",
"DispatchPayload" : null,
"VolumeMounts" : null,
"KillSignal" : "",
"Driver" : "docker",
"LogConfig" : {
"MaxFileSizeMB" : 1,
"MaxFiles" : 10
},
"Name" : "group1Task"
}
],
"Volumes" : null,
"Migrate" : null,
"Scaling" : null,
"EphemeralDisk" : {
"Migrate" : false,
"Sticky" : false,
"SizeMB" : 300
},
"Constraints" : null,
"StopAfterClientDisconnect" : null,
"Count" : 1,
"Spreads" : null,
"Name" : "grp_1"
},
... grp_2 to grp_42
],
"SubmitTime" : 1598412669315392718,
"Meta" : {
"owner" : "rnd",
"project" : "myProject"
},
"Type" : "batch",
"Update" : {
"MinHealthyTime" : 0,
"HealthCheck" : "",
"AutoPromote" : false,
"Stagger" : 0,
"MaxParallel" : 0,
"HealthyDeadline" : 0,
"AutoRevert" : false,
"Canary" : 0,
"ProgressDeadline" : 0
},
"VaultToken" : "",
"Region" : "us",
"Dispatched" : false,
"Payload" : null,
"ModifyIndex" : 227637,
"Periodic" : null,
"Constraints" : [
{
"LTarget" : "${node.class}",
"Operand" : "=",
"RTarget" : "default"
}
],
"AllAtOnce" : false,
"ParameterizedJob" : null,
"Version" : 0,
"StatusDescription" : "",
"Datacenters" : [
"hq"
],
"Namespace" : "default",
"ParentID" : "",
"Priority" : 50,
"Stop" : false,
"Spreads" : null,
"ConsulToken" : "",
"Name" : "myJob"
}
```
### Nomad Client logs (if appropriate)
Sadly, it cannot be retrieved.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the allocation behavior exposed by `nomad job status`, `nomad alloc restart`, and `nomad alloc stop`, then trace how a lost client node is handled for the affected task groups. Reproduce the client failure with the provided batch job and verify that lost allocations are represented consistently and can be reallocated or otherwise reach the expected terminal state.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go
- Domain
- devops, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100