task events missing or misleading around `shutdown_delay`
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
When a task has a `shutdown_delay`, we emit a "Waiting for shutdown delay" event at the start of the delay. Then when that delay expires we emit another event for "Killing" where we terminate the task. But this order is confusing because it looks like we're sending SIGINT much earlier than we should be. Example using a 30s task-level `shutdown_delay`:
```
Recent Events:
Time Type Description
2025-09-22T15:23:33-04:00 Killed Task successfully killed
2025-09-22T15:23:33-04:00 Terminated Exit Code: 137, Exit Message: "Docker container exited with non-zero exit code: 137"
2025-09-22T15:22:58-04:00 Waiting for shutdown delay Waiting for shutdown_delay of 30s before killing the task.
2025-09-22T15:22:58-04:00 Killing Sent interrupt. Waiting 5s before force killing
2025-09-22T15:21:41-04:00 Started Task started by client
2025-09-22T15:21:40-04:00 Task Setup Building Task Directory
2025-09-22T15:20:48-04:00 Received Task received by client
```
When `shutdown_delay` is set at the group level, we don't get the event at all, so the task events look like the allocation is just hung rather than pending shutdown, although it does shut down as we'd expect. Example using a 30s group-level `shutdown_delay`:
```
Recent Events:
Time Type Description
2025-09-22T15:40:50-04:00 Killed Task successfully killed
2025-09-22T15:40:50-04:00 Terminated Exit Code: 137, Exit Message: "Docker container exited with non-zero exit code: 137"
2025-09-22T15:40:45-04:00 Killing Sent interrupt. Waiting 5s before force killing
2025-09-22T15:39:04-04:00 Started Task started by client
2025-09-22T15:39:03-04:00 Task Setup Building Task Directory
2025-09-22T15:38:19-04:00 Received Task received by client
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Trace the task event emission paths for task-level and group-level shutdown_delay, starting with the shutdown sequence described in the examples. Compare when the waiting and killing events are recorded; done means the event timeline accurately reflects the delay before interruption and includes the applicable group-level delay.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100