hashicorp / hashicorp/nomad

worker.service_sched: processing eval panicked scheduler - please report this as a bug

Open
#20,157 3 comments 0 reactions 0 assignees View on GitHub
stage/needs-investigation theme/crash theme/scheduling type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version

Nomad v1.7.5
BuildDate 2024-02-13T15:10:13Z
Revision 5f5d4646198d09b8f4f6cb90fb5d50b53fa328b8

### Operating system and Environment details

RHEL 9.3

### Issue

Evaluations for the job are failing:

```
{
"priority": 50,
"type": "service",
"triggeredBy": "failed-follow-up",
"status": "failed",
"statusDescription": "evaluation reached delivery limit (3)",
"failedTGAllocs": [],
"previousEval": "34ab318b-a04d-a62b-48cc-604e265e4573",
"nextEval": "15f7b8cf-8091-5abf-f3ed-28517af63b7a",
"blockedEval": null,
"modifyIndex": 39197810,
"modifyTime": "2024-03-18T15:41:53.513Z",
"createIndex": 39197798,
"createTime": "2024-03-18T15:38:30.948Z",
"waitUntil": null,
"namespace": "default",
"plainJobId": "exec-job",
"relatedEvals": [
"15f7b8cf-8091-5abf-f3ed-28517af63b7a",
"34ab318b-a04d-a62b-48cc-604e265e4573",
"70e22606-6d2c-b44f-8062-b3a7b5f7ca69",
"6fa29ce2-9e16-2420-039e-7b5f8a4cd466",
"9f1bfb6c-4984-9b6d-384e-2defd5f1a574",
"7d362010-c877-74c9-56fc-b7b842688409",
"cc10963b-9de1-8d6b-ec1c-eaabf0f3497a",
"6503ee4a-5b86-d742-4597-cea96f18582e"
],
"job": "[\"exec-job\",\"default\"]",
"node": null
}
```

### Reproduction steps

Nomad cluster was updated to 1.7.5

#### Expected Result

Jobs are evaluated and running

#### Actual Result

Jobs are never started

### Job file (if appropriate)

Pretty much any job won't start.

### Nomad Server logs (if appropriate)

```
2024-03-18T15:25:14.668Z [ERROR] worker.service_sched: processing eval panicked scheduler - please report this as a bug!: eval_id=9f1bfb6c-4984-9b6d-384e-2defd5f1a574 job_id=exec-job namespace=default worker_id=0c9215c7-515a-eb81-7b10-a11f8abda944 eval_id=9f1bfb6c-4984-9b6d-384e-2defd5f1a574 error="runtime error: invalid memory address or nil pointer dereference"
stack_trace=
| goroutine 83 [running]:
| runtime/debug.Stack()
| \truntime/debug/stack.go:24 +0x5e
| github.com/hashicorp/nomad/scheduler.(*GenericScheduler).Process.func1()
| \tgithub.com/hashicorp/nomad/scheduler/generic_sched.go:153 +0x58
| panic({0x2a88140?, 0x4f5ea50?})
| \truntime/panic.go:914 +0x21f
| github.com/hashicorp/nomad/client/lib/numalib.(*Topology).UsableCores(...)
| \tgithub.com/hashicorp/nomad/client/lib/numalib/topology.go:258
| github.com/hashicorp/nomad/nomad/structs.(*NodeResources).Comparable(0xc001108c80)
| \tgithub.com/hashicorp/nomad/nomad/structs/structs.go:3185 +0xcc
| github.com/hashicorp/nomad/scheduler.(*Preemptor).SetNode(0xc0029c48f0, 0xc00cc18000)
| \tgithub.com/hashicorp/nomad/scheduler/preemption.go:139 +0x36
| github.com/hashicorp/nomad/scheduler.(*BinPackIterator).Next(0xc00c176a80)
| \tgithub.com/hashicorp/nomad/scheduler/rank.go:274 +0x74d
| github.com/hashicorp/nomad/scheduler.(*JobAntiAffinityIterator).Next(0xc00b367bd0)
| \tgithub.com/hashicorp/nomad/scheduler/rank.go:624 +0x6b
| github.com/hashicorp/nomad/scheduler.(*NodeReschedulingPenaltyIterator).Next(0xc00e4384e0)
| \tgithub.com/hashicorp/nomad/scheduler/rank.go:685 +0x28
| github.com/hashicorp/nomad/scheduler.(*NodeAffinityIterator).Next(0xc00b367c20)
| \tgithub.com/hashicorp/nomad/scheduler/rank.go:757 +0x30
| github.com/hashicorp/nomad/scheduler.(*SpreadIterator).Next(0xc00c176af0)
| \tgithub.com/hashicorp/nomad/scheduler/spread.go:131 +0x33
| github.com/hashicorp/nomad/scheduler.(*PreemptionScoringIterator).Next(0xc02e7cace0)
| \tgithub.com/hashicorp/nomad/scheduler/rank.go:852 +0x28
| github.com/hashicorp/nomad/scheduler.(*ScoreNormalizationIterator).Next(0xc02e7cad20)
| \tgithub.com/hashicorp/nomad/scheduler/rank.go:816 +0x28
| github.com/hashicorp/nomad/scheduler.(*LimitIterator).nextOption(0xc008a79aa0)
| \tgithub.com/hashicorp/nomad/scheduler/select.go:63 +0x24
| github.com/hashicorp/nomad/scheduler.(*LimitIterator).Next(0xc008a79aa0)
| \tgithub.com/hashicorp/nomad/scheduler/select.go:42 +0x26
| github.com/hashicorp/nomad/scheduler.(*MaxScoreIterator).Next(0xc00e438570)
| \tgithub.com/hashicorp/nomad/scheduler/select.go:105 +0x3e
| github.com/hashicorp/nomad/scheduler.(*GenericStack).Select(0xc0262d92b0, 0xc00c062b40, 0xc0029c5530)
| \tgithub.com/hashicorp/nomad/scheduler/stack.go:192 +0xe8f
| github.com/hashicorp/nomad/scheduler.(*GenericScheduler).selectNextOption(0xc00985c000, 0x38264a0?, 0xc0029c5530)
| \tgithub.com/hashicorp/nomad/scheduler/generic_sched.go:898 +0x2d
| github.com/hashicorp/nomad/scheduler.(*GenericScheduler).computePlacements(0xc00985c000, {0x526ef20, 0x0, 0x0}, {0xc00b5c5740, 0x1, 0x1}, 0x0?)
| \tgithub.com/hashicorp/nomad/scheduler/generic_sched.go:602 +0xa47
| github.com/hashicorp/nomad/scheduler.(*GenericScheduler).computeJobAllocs(0xc00985c000)
| \tgithub.com/hashicorp/nomad/scheduler/generic_sched.go:469 +0x14da
| github.com/hashicorp/nomad/scheduler.(*GenericScheduler).process(0xc00985c000)
| \tgithub.com/hashicorp/nomad/scheduler/generic_sched.go:289 +0x49a
| github.com/hashicorp/nomad/scheduler.retryMax(0x5, 0xc0029c5d20, 0xc0029c5d10)
| \tgithub.com/hashicorp/nomad/scheduler/util.go:96 +0x49
| github.com/hashicorp/nomad/scheduler.(*GenericScheduler).Process(0xc00985c000, 0xc01c1d7680)
| \tgithub.com/hashicorp/nomad/scheduler/generic_sched.go:188 +0x55f
| github.com/hashicorp/nomad/nomad.(*Worker).invokeScheduler(0xc008e70ee0, 0xc0110d1e60, 0xc01c1d7680, {0xc02005da10, 0x24})
| \tgithub.com/hashicorp/nomad/nomad/worker.go:634 +0x353
| github.com/hashicorp/nomad/nomad.(*Worker).run(0xc008e70ee0, 0x12a05f200)
| \tgithub.com/hashicorp/nomad/nomad/worker.go:463 +0x5a5
| created by github.com/hashicorp/nomad/nomad.(*Worker).Start in goroutine 1
| \tgithub.com/hashicorp/nomad/nomad/worker.go:162 +0x59
```

### Nomad Client logs (if appropriate)

N/A

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with client/lib/numalib/topology.go at Topology.UsableCores and follow NodeResources.Comparable in nomad/structs/structs.go, then inspect the scheduler path through preemption.go shown in the stack trace. Reproduce the evaluation failure if possible and verify that scheduling no longer panics and jobs are evaluated and started.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.