nomad job restart command should stop leader task first

Open

#19,403 opened on Dec 9, 2023 2 comments 1 reaction 0 assignees View on GitHub

hcc/cst stage/accepted theme/restart/reschedule type/bug

Dominant language: Go
Stars: 17k
Forks: 2.1k
Avg merge: 1d 9h
Merged PRs (30d): 105

Description

### Nomad version
Output from `nomad version`

```
vagrant@linux:/opt/gopath/src/github.com/hashicorp/nomad$ nomad version
Nomad v1.7.0-dev+ent
BuildDate 2023-11-02T04:09:19Z
Revision 02cf9f45545e3f57022d6a8661ae32eb991863dc
```

### Operating system and Environment details

```
vagrant@linux:/opt/gopath/src/github.com/hashicorp/nomad$ uname -a
Linux linux 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
```

### Issue

When initiating a job restart command the tasks are restarted without applying lifecycle rules (pre-start tasks first...etc) nor leader flag.

### Reproduction steps

have one task groups with prestart tasks, leader tasks and shutdown_delay tasks.

restart the job eg.

`nomad job restart sleep-while-lifecycle`

#### Expected Result

The lifecycle of the task group should be honored

#### Actual Result

Leader and sidecar tasks are sent a signal at the same time

### Job file (if appropriate)

```
job sleep-while-lifecycle {
datacenters = ["dc1"]
group "group" {
count = 1

## You might want to constrain this, so here's one to help
# constraint {
# attribute = "${attr.unique.hostname}"
# operator = "="
# value = "nomad-client-1.node.consul"
# }

task "leader" {
template {
data = < To: nomad-oss-debug@hashicorp.com
>
> Subject: GH-1234: Errors garbage collecting allocs

Emails sent to that address are readable by all HashiCorp employees but are
*not* publicly visible.
-->

### Nomad Server logs (if appropriate)

I’ve added the log outputs from the tasks and you can see they receive the signal all at the same time when I initiate a job restart `Fri Dec 1 18:24:54 UTC 2023 - Starting. SLEEP_SECS=2`

```
vagrant@linux:/opt/gopath/src/github.com/hashicorp/nomad$ nomad job restart sleep-while-lifecycle
==> 2023-12-01T18:24:49Z: Restarting 1 allocation
2023-12-01T18:24:49Z: Restarting running tasks in allocation "868d67ca" for group "group"
==> 2023-12-01T18:24:54Z: Job restart finished

Job restarted successfully!
```

```
leader task stdout

Fri Dec 1 18:23:56 UTC 2023 - Starting. SLEEP_SECS=2
Fri Dec 1 18:23:56 UTC 2023 - Sleeping for 2 seconds.
Fri Dec 1 18:23:58 UTC 2023 - Sleeping for 2 seconds.
Fri Dec 1 18:24:00 UTC 2023 - Sleeping for 2 seconds.
Fri Dec 1 18:24:02 UTC 2023 - Sleeping for 2 seconds.
Fri Dec 1 18:24:04 UTC 2023 - Sleeping for 2 seconds.
Fri Dec 1 18:24:06 UTC 2023 - Sleeping for 2 seconds.
Fri Dec 1 18:24:08 UTC 2023 - Sleeping for 2 seconds.
Fri Dec 1 18:24:10 UTC 2023 - Sleeping for 2 seconds.
Fri Dec 1 18:24:12 UTC 2023 - Sleeping for 2 seconds.
Fri Dec 1 18:24:14 UTC 2023 - Sleeping for 2 seconds.
Fri Dec 1 18:24:16 UTC 2023 - Sleeping for 2 seconds.
Fri Dec 1 18:24:18 UTC 2023 - Sleeping for 2 seconds.
Fri Dec 1 18:24:20 UTC 2023 - Sleeping for 2 seconds.
Fri Dec 1 18:24:22 UTC 2023 - Sleeping for 2 seconds.
Fri Dec 1 18:24:24 UTC 2023 - Sleeping for 2 seconds.
Fri Dec 1 18:24:26 UTC 2023 - Sleeping for 2 seconds.
Fri Dec 1 18:24:28 UTC 2023 - Sleeping for 2 seconds.
Fri Dec 1 18:24:30 UTC 2023 - Sleeping for 2 seconds.
Fri Dec 1 18:24:32 UTC 2023 - Sleeping for 2 seconds.
Fri Dec 1 18:24:34 UTC 2023 - Sleeping for 2 seconds.
Fri Dec 1 18:24:36 UTC 2023 - Sleeping for 2 seconds.
Fri Dec 1 18:24:38 UTC 2023 - Sleeping for 2 seconds.
Fri Dec 1 18:24:40 UTC 2023 - Sleeping for 2 seconds.
Fri Dec 1 18:24:42 UTC 2023 - Sleeping for 2 seconds.
Fri Dec 1 18:24:44 UTC 2023 - Sleeping for 2 seconds.
Fri Dec 1 18:24:46 UTC 2023 - Sleeping for 2 seconds.
Fri Dec 1 18:24:48 UTC 2023 - Sleeping for 2 seconds.
Fri Dec 1 18:24:50 UTC 2023 - Sleeping for 2 seconds.
Fri Dec 1 18:24:52 UTC 2023 - Sleeping for 2 seconds.
Fri Dec 1 18:24:54 UTC 2023 - Starting. SLEEP_SECS=2
Fri Dec 1 18:24:54 UTC 2023 - Sleeping for 2 seconds.
Fri Dec 1 18:24:56 UTC 2023 - Sleeping for 2 seconds.

sidecar task stdout

regular task stdout

### Nomad Client logs (if appropriate)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the `nomad job restart sleep-while-lifecycle` entry point and reproduce the behavior using the job file and task logs provided. Confirm that restart ordering honors pre-start tasks, the leader flag, sidecars, regular tasks, and shutdown delays; the issue is done when these tasks no longer receive restart signals simultaneously.

Written by the indexing model from the issue text.

Assessment

Tech stack: go
Domain: cli
Issue type: Bug
Difficulty: 4/5
Estimated time: 3-5 days
Activity status: Stale
Clarity: Mostly clear
Newbie friendliness: 38/100

nomad job restart command should stop leader task first

Description

Contributor guide

Research direction

Assessment

Get new issues in your inbox