hashicorp / hashicorp/nomad

TaskGroup ends in Queued & Complete state after client node lost

Open
#8,782 4 comments 0 reactions 0 assignees View on GitHub
stage/needs-investigation theme/restart/reschedule type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
Nomad v0.11.4 (3c6abec8d4a5c1af96ac3e1910a90949390dea19)

### Operating system and Environment details
CentOS 7.8.2003 on bare-metal

### Issue
I have a batch job with 42 individual tasks wrapped in 42 groups. During run, one of the client nodes failed and left the Nomad cluster altogether. If I run `nomad job status` on this job, the 2 groups that landed on that failed node show a summary state of one "Queued" and one "Complete". Like so,

```
ID = gmx-covid19_complex_tmprss2-5505-ps1_2020-08-25-0
Name = gmx-covid19_complex_tmprss2-5505-ps1_2020-08-25-0
Submit Date = 2020-08-25T20:31:09-07:00
Type = batch
Priority = 50
Datacenters = hq
Namespace = default
Status = running
Periodic = false
Parameterized = false

Summary
Task Group Queued Starting Running Failed Complete Lost
grp_1 0 0 1 0 0 0
...
grp_13 1 0 0 0 1 0 <= NOTE: Queued & Complete
...
grp_17 1 0 0 0 1 0 <= NOTE: Queued & Complete
...

Allocations
ID Node ID Task Group Version Desired Status Created Modified
792be1a8 399a6eac grp_1 0 run running 10h35m ago 10h35m ago
...
9e194fb5 0566cec4 grp_13 0 stop complete 2d15h ago 2d6m ago <= NOTE: stop & complete
...
1d6be2ca 0566cec4 grp_17 0 stop complete 2d18h ago 2d6m ago <= NOTE: stop & complete
```

(`grp_1` is included as an example of normal.)

Since the failed node is no longer part of the Nomad cluster, `nomad alloc restart ` failed with a 500 error stating the node doesn't exist. If instead I issue a `nomad alloc stop ` on these 2 allocations, no error occurs, but the "Queued" states changed to "0".

From what I can see, the tasks are lost and it does not appear Nomad will reallocate them elsewhere.
So here are 2 issues:

* Why are the groups identified as "Queued"? It does not appear they will be reallocated.
* It would be very useful to be able to force Nomad to reallocate these lost groups.

### Reproduction steps
I'm afraid you'd have to take out a client node while the allocations are running to reproduce this.

### Job file (if appropriate)
This is the trimmed output of `curl http://127.0.0.1:4646/v1/job/myJob`:

```
{
"Stable" : false,
"ID" : "myJob",
"CreateIndex" : 226771,
"Affinities" : null,
"Status" : "running",
"JobModifyIndex" : 226771,
"TaskGroups" : [
{
"Affinities" : null,
"ReschedulePolicy" : {
"Interval" : 86400000000000,
"Attempts" : 1,
"Unlimited" : false,
"MaxDelay" : 0,
"DelayFunction" : "constant",
"Delay" : 5000000000
},
"ShutdownDelay" : null,
"Meta" : null,
"RestartPolicy" : {
"Interval" : 86400000000000,
"Attempts" : 25,
"Mode" : "fail",
"Delay" : 1800000000000
},
"Networks" : null,
"Services" : null,
"Update" : null,
"Tasks" : [
{
"Affinities" : null,
"Templates" : null,
"CSIPluginConfig" : null,
"ShutdownDelay" : 0,
"User" : "",
"Config" : {
"volumes" : [
"/mnt/shared/myJob/input:/run.d/input:ro",
"/mnt/shared/myJob/output:/run.d/output",
],
"args" : [
"--verbose",
"--uid=1000043",
"--gid=100"
],
"image" : "myJob/image:1",
"network_mode" : "none"
},
"Leader" : false,
"Meta" : null,
"RestartPolicy" : {
"Interval" : 86400000000000,
"Attempts" : 25,
"Mode" : "fail",
"Delay" : 1800000000000
},
"Resources" : {
"Networks" : null,
"Devices" : null,
"IOPS" : 0,
"DiskMB" : 0,
"CPU" : 20000,
"MemoryMB" : 4000
},
"Services" : null,
"Vault" : null,
"KillTimeout" : 20000000000,
"Artifacts" : null,
"Constraints" : null,
"Lifecycle" : null,
"Env" : {
"SYSTEM" : "complex",
"GROUP" : "group1"
},
"Kind" : "",
"DispatchPayload" : null,
"VolumeMounts" : null,
"KillSignal" : "",
"Driver" : "docker",
"LogConfig" : {
"MaxFileSizeMB" : 1,
"MaxFiles" : 10
},
"Name" : "group1Task"
}
],
"Volumes" : null,
"Migrate" : null,
"Scaling" : null,
"EphemeralDisk" : {
"Migrate" : false,
"Sticky" : false,
"SizeMB" : 300
},
"Constraints" : null,
"StopAfterClientDisconnect" : null,
"Count" : 1,
"Spreads" : null,
"Name" : "grp_1"
},
... grp_2 to grp_42
],
"SubmitTime" : 1598412669315392718,
"Meta" : {
"owner" : "rnd",
"project" : "myProject"
},
"Type" : "batch",
"Update" : {
"MinHealthyTime" : 0,
"HealthCheck" : "",
"AutoPromote" : false,
"Stagger" : 0,
"MaxParallel" : 0,
"HealthyDeadline" : 0,
"AutoRevert" : false,
"Canary" : 0,
"ProgressDeadline" : 0
},
"VaultToken" : "",
"Region" : "us",
"Dispatched" : false,
"Payload" : null,
"ModifyIndex" : 227637,
"Periodic" : null,
"Constraints" : [
{
"LTarget" : "${node.class}",
"Operand" : "=",
"RTarget" : "default"
}
],
"AllAtOnce" : false,
"ParameterizedJob" : null,
"Version" : 0,
"StatusDescription" : "",
"Datacenters" : [
"hq"
],
"Namespace" : "default",
"ParentID" : "",
"Priority" : 50,
"Stop" : false,
"Spreads" : null,
"ConsulToken" : "",
"Name" : "myJob"
}
```

### Nomad Client logs (if appropriate)

Sadly, it cannot be retrieved.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the allocation behavior exposed by `nomad job status`, `nomad alloc restart`, and `nomad alloc stop`, then trace how a lost client node is handled for the affected task groups. Reproduce the client failure with the provided batch job and verify that lost allocations are represented consistently and can be reallocated or otherwise reach the expected terminal state.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
devops, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.