hashicorp / hashicorp/nomad

System job not restarting after client failure.

Open
#15,069 7 comments 0 reactions 0 assignees View on GitHub
stage/needs-verification theme/system-scheduler theme/template type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
1.4.2

### Operating system and Environment details
Ubuntu arm64

### Issue
If the client is down, system job on that client is not restarted unless manually done.
Screenshot 2022-10-28 at 1 51 09 PM
Also the job status in client shows 2 failed. It should be `1 failed 1 passed` because as you can see below, there is one job running.
### Reproduction steps
Run a system job. Take the client (or the whole cluster ?) down. Bring the nodes up. Check if the system job has all allocations.

#### Expected Result
All allocations present.

#### Actual Result
Not all allocations present.

### Job file (if appropriate)
Same as https://github.com/hashicorp/nomad/issues/14932

### Nomad Server logs (if appropriate)

Could see the alloc was killed due to
```
Template failed: nomad.var.get(nomad/jobs/caddy/caddy/caddy@default.global): Unexpected response code: 500 (rpc error: failed to get conn: rpc error: lead thread didn't get connection)
```

### Nomad Client logs (if appropriate)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the reproduction steps for a Nomad system job on Ubuntu arm64 and compare this report with issue #14932. Reproduce the client shutdown and restart, then inspect the allocation and job status behavior; done means all system-job allocations return and the status reports one failed and one passed as expected.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.