Unable to Start Job in Nomad GUI After Entering "Dead" State
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
Hello! We're having an issue starting jobs again from the UI after they crash.
### Nomad version
```
# nomad version
Nomad v1.5.1
BuildDate 2023-03-10T22:05:57Z
Revision 6c118ddaf95e9bfd3d85f488f255fdc5e14129c7
```
### Operating system and Environment details
Docker
```
# uname -a
Linux ip-10-1-23-220.ec2.internal 5.15.0-52-generic #58-Ubuntu SMP Thu Oct 13 08:03:55 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
```
### Issue
When a Nomad job enters the "dead" state after crashing / failing, attempting to start the job again from the Nomad GUI does not result in the job launching successfully. Despite clicking the "start" button and seeing the job transition to the "pending" state, the job remains inactive and returns to the "dead" state after refreshing the GUI.
### Reproduction steps
1. Deploy a Nomad job to the cluster.
2. Allow the job to run and eventually encounter a situation where it enters the "dead" state (e.g., due to a service crash or reaching the restart limit).
3. Navigate to the Nomad GUI and locate the job in the list of jobs.
4. Click the "start" button next to the job to initiate a restart.
#### Expected Result
After clicking the "start" button, the job should be relaunched and transition to the "running" or "pending" state successfully, regardless of why or how it stopped.
#### Actual Result
Upon clicking the "start" button, the job appears to transition to the "pending" state but never actually starts. Upon refreshing the GUI, the job is still in the "dead" state.
### Workaround
We found a workaround by modifying the Nomad job specification. By setting the "stop" property to "true" in the job specification, it becomes possible to start the job from the Nomad GUI, and the job launches as expected. Another workaround is to trigger a new deployment by materially changing the job definition (e.g. adding an environment variable, changing a template, etc.), but this is less desirable because it causes `nomad job plan` to show differences when run with the original job spec, breaking idempotency.
### Job file (if appropriate)
Service job (can't share file).
### Nomad Server logs (if appropriate)
Not captured.
### Nomad Client logs (if appropriate)
Not captured.
If it would be helpful I can try to capture server/client logs while clicking the "start" button.
Thanks!
-Kevin
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the failure through the Nomad GUI with a job in the dead state, while capturing the Nomad server and client logs that were not included. The issue is resolved when clicking Start relaunches the job and it remains in a running or pending state after refreshing the GUI.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, linux
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100