Client panics after many allocations are stopped
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
Nomad v1.10.0
BuildDate 2025-04-09T16:40:54Z
Revision e26a2bd2acac2dcdcb623f4d293bac096beef478
### Operating system and Environment details
Rocky 9.3
### Issue
Client crashes after stopping many allocations. This started after upgrading to v1.10.0 from v1.8.4
### Reproduction steps
Stop many allocations
#### Expected Result
Allocations are rescheduled
#### Actual Result
Client panics
### Nomad Client logs (if appropriate)
-- VISUAL LINE --731,1Top
May 06 23:56:59 nomad[3795872]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x21d1d15]May 06 23:56:59 nomad[3795872]: goroutine 1022 [running]:
May 06 23:56:59 nomad[3795872]: github.com/hashicorp/nomad/client/allocrunner.(*observer).stop(...)May 06 23:56:59 nomad[3795872]: github.com/hashicorp/nomad/client/allocrunner/checks_hook.go:73May 06 23:56:59 nomad[3795872]: github.com/hashicorp/nomad/client/allocrunner.(*checksHook).Update(0xc001262510, 0xc005967c40)
May 06 23:56:59 nomad[3795872]: github.com/hashicorp/nomad/client/allocrunner/checks_hook.go:256 +0x275May 06 23:56:59 nomad[3795872]: github.com/hashicorp/nomad/client/allocrunner.(*allocRunner).update(0xc00071d608, 0xc006c84e00)May 06 23:56:59 nomad[3795872]: github.com/hashicorp/nomad/client/allocrunner/alloc_runner_hooks.go:243 +0x4d7
May 06 23:56:59 nomad[3795872]: github.com/hashicorp/nomad/client/allocrunner.(*allocRunner).handleAllocUpdate(0xc00071d608, 0xc006c84e00)May 06 23:56:59 nomad[3795872]:github.com/hashicorp/nomad/client/allocrunner/alloc_runner.go:1050 +0x2b5May 06 23:56:59 nomad[3795872]: github.com/hashicorp/nomad/client/allocrunner.(*allocRunner).handleAllocUpdates(...)May 06 23:56:59 nomad[3795872]: github.com/hashicorp/nomad/client/allocrunner/alloc_runner.go:1031 May 06 23:56:59 nomad[3795872]: created by github.com/hashicorp/nomad/client/allocrunner.(*allocRunner).Run in goroutine 1186May 06 23:56:59 nomad[3795872]: github.com/hashicorp/nomad/client/allocrunner/alloc_runner.go:380 +0xdf May 06 23:56:59 systemd[1]: nomad.service: Main process exited, code=exited, status=2/INVALIDARGUMENTMay 06 23:56:59 systemd[1]: nomad.service: Failed with result 'exit-code'.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the panic stack in client/allocrunner/checks_hook.go, especially observer.stop and checksHook.Update around lines 73 and 256. Trace how allocRunner.update and handleAllocUpdates in alloc_runner.go process stopped allocations, then reproduce by stopping many allocations. Done means the client no longer panics and allocations are rescheduled as expected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100