hashicorp / hashicorp/nomad

task events missing or misleading around `shutdown_delay`

Open
#26,829 0 comments 0 reactions 0 assignees View on GitHub
hcc/jira theme/bad-ux theme/client type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

When a task has a `shutdown_delay`, we emit a "Waiting for shutdown delay" event at the start of the delay. Then when that delay expires we emit another event for "Killing" where we terminate the task. But this order is confusing because it looks like we're sending SIGINT much earlier than we should be. Example using a 30s task-level `shutdown_delay`:

```
Recent Events:
Time Type Description
2025-09-22T15:23:33-04:00 Killed Task successfully killed
2025-09-22T15:23:33-04:00 Terminated Exit Code: 137, Exit Message: "Docker container exited with non-zero exit code: 137"
2025-09-22T15:22:58-04:00 Waiting for shutdown delay Waiting for shutdown_delay of 30s before killing the task.
2025-09-22T15:22:58-04:00 Killing Sent interrupt. Waiting 5s before force killing
2025-09-22T15:21:41-04:00 Started Task started by client
2025-09-22T15:21:40-04:00 Task Setup Building Task Directory
2025-09-22T15:20:48-04:00 Received Task received by client
```

When `shutdown_delay` is set at the group level, we don't get the event at all, so the task events look like the allocation is just hung rather than pending shutdown, although it does shut down as we'd expect. Example using a 30s group-level `shutdown_delay`:

```
Recent Events:
Time Type Description
2025-09-22T15:40:50-04:00 Killed Task successfully killed
2025-09-22T15:40:50-04:00 Terminated Exit Code: 137, Exit Message: "Docker container exited with non-zero exit code: 137"
2025-09-22T15:40:45-04:00 Killing Sent interrupt. Waiting 5s before force killing
2025-09-22T15:39:04-04:00 Started Task started by client
2025-09-22T15:39:03-04:00 Task Setup Building Task Directory
2025-09-22T15:38:19-04:00 Received Task received by client
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Trace the task event emission paths for task-level and group-level shutdown_delay, starting with the shutdown sequence described in the examples. Compare when the waiting and killing events are recorded; done means the event timeline accurately reflects the delay before interruption and includes the applicable group-level delay.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.