hashicorp / hashicorp/nomad

Client panics after many allocations are stopped

Open
#25,820 3 comments 0 reactions 0 assignees View on GitHub
hcc/jira theme/crash type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
Nomad v1.10.0
BuildDate 2025-04-09T16:40:54Z
Revision e26a2bd2acac2dcdcb623f4d293bac096beef478

### Operating system and Environment details
Rocky 9.3

### Issue
Client crashes after stopping many allocations. This started after upgrading to v1.10.0 from v1.8.4

### Reproduction steps
Stop many allocations

#### Expected Result
Allocations are rescheduled

#### Actual Result
Client panics

### Nomad Client logs (if appropriate)
-- VISUAL LINE --731,1Top
May 06 23:56:59 nomad[3795872]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x21d1d15]May 06 23:56:59 nomad[3795872]: goroutine 1022 [running]:
May 06 23:56:59 nomad[3795872]: github.com/hashicorp/nomad/client/allocrunner.(*observer).stop(...)May 06 23:56:59 nomad[3795872]: github.com/hashicorp/nomad/client/allocrunner/checks_hook.go:73May 06 23:56:59 nomad[3795872]: github.com/hashicorp/nomad/client/allocrunner.(*checksHook).Update(0xc001262510, 0xc005967c40)
May 06 23:56:59 nomad[3795872]: github.com/hashicorp/nomad/client/allocrunner/checks_hook.go:256 +0x275May 06 23:56:59 nomad[3795872]: github.com/hashicorp/nomad/client/allocrunner.(*allocRunner).update(0xc00071d608, 0xc006c84e00)May 06 23:56:59 nomad[3795872]: github.com/hashicorp/nomad/client/allocrunner/alloc_runner_hooks.go:243 +0x4d7
May 06 23:56:59 nomad[3795872]: github.com/hashicorp/nomad/client/allocrunner.(*allocRunner).handleAllocUpdate(0xc00071d608, 0xc006c84e00)May 06 23:56:59 nomad[3795872]:github.com/hashicorp/nomad/client/allocrunner/alloc_runner.go:1050 +0x2b5May 06 23:56:59 nomad[3795872]: github.com/hashicorp/nomad/client/allocrunner.(*allocRunner).handleAllocUpdates(...)May 06 23:56:59 nomad[3795872]: github.com/hashicorp/nomad/client/allocrunner/alloc_runner.go:1031 May 06 23:56:59 nomad[3795872]: created by github.com/hashicorp/nomad/client/allocrunner.(*allocRunner).Run in goroutine 1186May 06 23:56:59 nomad[3795872]: github.com/hashicorp/nomad/client/allocrunner/alloc_runner.go:380 +0xdf May 06 23:56:59 systemd[1]: nomad.service: Main process exited, code=exited, status=2/INVALIDARGUMENTMay 06 23:56:59 systemd[1]: nomad.service: Failed with result 'exit-code'.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the panic stack in client/allocrunner/checks_hook.go, especially observer.stop and checksHook.Update around lines 73 and 256. Trace how allocRunner.update and handleAllocUpdates in alloc_runner.go process stopped allocations, then reproduce by stopping many allocations. Done means the client no longer panics and allocations are rescheduled as expected.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.