System job not restarting after client failure.
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
1.4.2
### Operating system and Environment details
Ubuntu arm64
### Issue
If the client is down, system job on that client is not restarted unless manually done.
Also the job status in client shows 2 failed. It should be `1 failed 1 passed` because as you can see below, there is one job running.
### Reproduction steps
Run a system job. Take the client (or the whole cluster ?) down. Bring the nodes up. Check if the system job has all allocations.
#### Expected Result
All allocations present.
#### Actual Result
Not all allocations present.
### Job file (if appropriate)
Same as https://github.com/hashicorp/nomad/issues/14932
### Nomad Server logs (if appropriate)
Could see the alloc was killed due to
```
Template failed: nomad.var.get(nomad/jobs/caddy/caddy/caddy@default.global): Unexpected response code: 500 (rpc error: failed to get conn: rpc error: lead thread didn't get connection)
```
### Nomad Client logs (if appropriate)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the reproduction steps for a Nomad system job on Ubuntu arm64 and compare this report with issue #14932. Reproduce the client shutdown and restart, then inspect the allocation and job status behavior; done means all system-job allocations return and the status reports one failed and one passed as expected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100