hashicorp / hashicorp/nomad

Interrupt sent to allocs that have long terminated

Open
#24,630 13 comments 0 reactions 0 assignees View on GitHub
hcc/jira theme/restart/reschedule type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
Nomad v1.9.3
BuildDate 2024-11-11T16:35:41Z
Revision d92bf1014886c0ff9f882f4a2691d5ae8ad8131c

### Operating system and Environment details
Ubuntu 22.04.5 LTS on the server.

Ubuntu 24.04.1 LTS on the client.

### Issue
Nomad clients send an interrupt on allocs that have long been terminated (but not yet gc'ed because of the settings).

### Reproduction steps
- Have some allocs (periodic / non-periodic)
- Keep those allocs around
- `job_gc_threshold = "24h"`
- Run the GC

#### Expected Result
Tasks that have long terminated to not receive any signal / interrupt / event. `Terminated` should be a terminal state.

#### Actual Result
![Screenshot_20241209_121455](https://github.com/user-attachments/assets/b5a8c40c-1348-42b4-8b87-fd9c14194fdd)

### Job file (if appropriate)
Probably irrelevant, although I did set `shutdown_delay: 10s` on the group.

### Nomad Server logs (if appropriate)
Nothing related in that timespan.

### Nomad Client logs (if appropriate)
```
2024-12-09T11:20:33.049Z [TRACE] client.alloc_runner.task_runner: Kill requested: alloc_id=3218e2e3-9973-b587-d518-ba9564720e23 task=surrealdb
2024-12-09T11:20:33.049Z [TRACE] client.alloc_runner.task_runner: Kill event: alloc_id=3218e2e3-9973-b587-d518-ba9564720e23 task=surrealdb event_type=Killing event_reason=""
2024-12-09T11:20:33.050Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=3218e2e3-9973-b587-d518-ba9564720e23 task=surrealdb type=Killing msg="Sent interrupt. Waiting 5s before force killing" failed=false
2024-12-09T11:20:33.052Z [TRACE] client.alloc_runner.task_runner: Kill requested: alloc_id=3218e2e3-9973-b587-d518-ba9564720e23 task=connect-proxy-surrealdb
2024-12-09T11:20:33.052Z [TRACE] client.alloc_runner.task_runner: Kill event: alloc_id=3218e2e3-9973-b587-d518-ba9564720e23 task=connect-proxy-surrealdb event_type=Killing event_reason=""
2024-12-09T11:20:33.052Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=3218e2e3-9973-b587-d518-ba9564720e23 task=connect-proxy-surrealdb type=Killing msg="Sent interrupt. Waiting 5s before force killing" failed=false
2024-12-09T11:20:33.054Z [INFO] client.gc: marking allocation for GC: alloc_id=3218e2e3-9973-b587-d518-ba9564720e23
2024-12-09T11:20:33.056Z [DEBUG] client.gc: alloc garbage collected: alloc_id=3218e2e3-9973-b587-d518-ba9564720e23
2024-12-09T11:20:33.057Z [INFO] client.gc: garbage collecting allocation: alloc_id=af2223c2-87e9-7358-b847-f68fc05eca19 reason="number of allocations (184) is over the limit (50)"
2024-12-09T11:20:33.057Z [DEBUG] client.alloc_runner.runner_hook.group_services: delay before killing tasks: alloc_id=af2223c2-87e9-7358-b847-f68fc05eca19 group=sync shutdown_delay=10s
2024-12-09T11:20:33.619Z [TRACE] client: next heartbeat: period=19.809518426s
2024-12-09T11:20:33.705Z [DEBUG] client: updated allocations: index=4549798 total=347 pulled=1 filtered=346
2024-12-09T11:20:33.705Z [DEBUG] client: allocation updates: added=0 removed=0 updated=1 ignored=346
2024-12-09T11:20:33.707Z [TRACE] client.alloc_runner: AllocRunner has terminated, skipping alloc update: alloc_id=ad3504fc-c1af-2abb-d0fb-6b5f6746b56c modify_index=4549749
2024-12-09T11:20:33.707Z [DEBUG] client: allocation updates applied: added=0 removed=0 updated=1 ignored=346 errors=0
2024-12-09T11:20:33.719Z [TRACE] client: next heartbeat: period=13.938668795s
```

Potentially related to https://github.com/hashicorp/nomad/issues/19917

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the client.alloc_runner and client.gc paths shown in the logs, then reproduce the issue using retained periodic or non-periodic allocations and job_gc_threshold="24h". Trace why a terminated allocation receives a kill event during GC; done means terminated allocations receive no further signal, interrupt, or event before or during garbage collection.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.