Interrupt sent to allocs that have long terminated
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
Nomad v1.9.3
BuildDate 2024-11-11T16:35:41Z
Revision d92bf1014886c0ff9f882f4a2691d5ae8ad8131c
### Operating system and Environment details
Ubuntu 22.04.5 LTS on the server.
Ubuntu 24.04.1 LTS on the client.
### Issue
Nomad clients send an interrupt on allocs that have long been terminated (but not yet gc'ed because of the settings).
### Reproduction steps
- Have some allocs (periodic / non-periodic)
- Keep those allocs around
- `job_gc_threshold = "24h"`
- Run the GC
#### Expected Result
Tasks that have long terminated to not receive any signal / interrupt / event. `Terminated` should be a terminal state.
#### Actual Result

### Job file (if appropriate)
Probably irrelevant, although I did set `shutdown_delay: 10s` on the group.
### Nomad Server logs (if appropriate)
Nothing related in that timespan.
### Nomad Client logs (if appropriate)
```
2024-12-09T11:20:33.049Z [TRACE] client.alloc_runner.task_runner: Kill requested: alloc_id=3218e2e3-9973-b587-d518-ba9564720e23 task=surrealdb
2024-12-09T11:20:33.049Z [TRACE] client.alloc_runner.task_runner: Kill event: alloc_id=3218e2e3-9973-b587-d518-ba9564720e23 task=surrealdb event_type=Killing event_reason=""
2024-12-09T11:20:33.050Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=3218e2e3-9973-b587-d518-ba9564720e23 task=surrealdb type=Killing msg="Sent interrupt. Waiting 5s before force killing" failed=false
2024-12-09T11:20:33.052Z [TRACE] client.alloc_runner.task_runner: Kill requested: alloc_id=3218e2e3-9973-b587-d518-ba9564720e23 task=connect-proxy-surrealdb
2024-12-09T11:20:33.052Z [TRACE] client.alloc_runner.task_runner: Kill event: alloc_id=3218e2e3-9973-b587-d518-ba9564720e23 task=connect-proxy-surrealdb event_type=Killing event_reason=""
2024-12-09T11:20:33.052Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=3218e2e3-9973-b587-d518-ba9564720e23 task=connect-proxy-surrealdb type=Killing msg="Sent interrupt. Waiting 5s before force killing" failed=false
2024-12-09T11:20:33.054Z [INFO] client.gc: marking allocation for GC: alloc_id=3218e2e3-9973-b587-d518-ba9564720e23
2024-12-09T11:20:33.056Z [DEBUG] client.gc: alloc garbage collected: alloc_id=3218e2e3-9973-b587-d518-ba9564720e23
2024-12-09T11:20:33.057Z [INFO] client.gc: garbage collecting allocation: alloc_id=af2223c2-87e9-7358-b847-f68fc05eca19 reason="number of allocations (184) is over the limit (50)"
2024-12-09T11:20:33.057Z [DEBUG] client.alloc_runner.runner_hook.group_services: delay before killing tasks: alloc_id=af2223c2-87e9-7358-b847-f68fc05eca19 group=sync shutdown_delay=10s
2024-12-09T11:20:33.619Z [TRACE] client: next heartbeat: period=19.809518426s
2024-12-09T11:20:33.705Z [DEBUG] client: updated allocations: index=4549798 total=347 pulled=1 filtered=346
2024-12-09T11:20:33.705Z [DEBUG] client: allocation updates: added=0 removed=0 updated=1 ignored=346
2024-12-09T11:20:33.707Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=ad3504fc-c1af-2abb-d0fb-6b5f6746b56c modify_index=4549749
2024-12-09T11:20:33.707Z [DEBUG] client: allocation updates applied: added=0 removed=0 updated=1 ignored=346 errors=0
2024-12-09T11:20:33.719Z [TRACE] client: next heartbeat: period=13.938668795s
```
Potentially related to https://github.com/hashicorp/nomad/issues/19917
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the client.alloc_runner and client.gc paths shown in the logs, then reproduce the issue using retained periodic or non-periodic allocations and job_gc_threshold="24h". Trace why a terminated allocation receives a kill event during GC; done means terminated allocations receive no further signal, interrupt, or event before or during garbage collection.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100