Nomad leader resource exhaustion when system jobs fail placement
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
Nomad v1.6.2
BuildDate: 2023-09-13
Revision: 73e372ad94033db2ceaf53468b270a31544c23fd
### Environment
Distributor ID: Ubuntu
Description: Ubuntu 22.04.5 LTS
Release: 22.04
Codename: jammy
### Issue
Nomad leader CPU and memory usage grows steadily and eventually becomes unhealthy when multiple historical **system jobs** repeatedly fail placement due to static port conflicts and prestart failures. The issue appeared after adding a `network` block to the job specification and required full cluster recreation to recover.
### Description
We deploy a system service that should run on all worker nodes. To avoid replacing already-running allocations during upgrades, each deployment creates a new system job with a unique job ID. Older job versions are prevented from running on newly deployed nodes via a prestart check, and allocations on nodes already running the service fail due to static port conflicts.
After introducing a `network` block with static ports, we observed the following behavior:
- Nomad leader CPU and memory usage steadily increased over time
- The leader eventually stopped responding to health checks
- After leader restart, the new leader exhibited the same behavior
- Recovery required recreating the entire Nomad cluster
At the time of failure, there were many historical system jobs present simultaneously, each continuing to be evaluated against all client nodes and failing placement.
### Observed behavior
- Sustained growth in Nomad leader CPU and memory
- Large volume of failed evaluations and allocations
- Leader becomes unhealthy and restarts
- Behavior repeats after leadership change
- No obvious recovery without cluster rebuild
### Suspected contributing factors
- Large number of historical system jobs targeting all nodes
- Static port conflicts causing immediate allocation failure
- Increased evaluation churn after adding the `network` block
- Possible unbounded evaluator or allocation state growth on the leader
### Reproduction
We were unable to reliably reproduce this in a smaller test cluster. The issue was observed only at the production scale, with many nodes and historical system jobs present.
### Job file (if appropriate)
```
job "service-${latest_job_id}" {
type = "system"
node_pool = "default"
priority = 90
group "service-client" {
network {
port "api" {
static = "5008"
}
port "proxy" {
static = "5007"
}
}
service {
name = "api"
port = "5008"
provider = "nomad"
check {
type = "http"
path = "/health"
name = "health"
interval = "20s"
timeout = "5s"
}
}
service {
name = "proxy"
port = "5007"
provider = "nomad"
check {
type = "tcp"
name = "health"
interval = "30s"
timeout = "1s"
}
}
task "check-placement" {
driver = "raw_exec"
lifecycle {
hook = "prestart"
sidecar = false
}
restart {
attempts = 0
}
template {
destination = "local/check-placement.sh"
data = <
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.