Nomad restart logs Error about previously failed canary deployment
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
Nomad v0.11.2 (807cfebe90d56f9e5beec3e72936ebe86acc8ce3)
### Operating system and Environment details
Debian Stable
### Issue
When restarting nomad I get the following error in the logs:
```
May 18 08:54:06 nomad01 nomad[16457]: 2020-05-18T08:54:00.161Z [ERROR] nomad.fsm: UpsertDeploymentPromotion failed: error="1 error occurred:
May 18 08:54:06 nomad01 nomad[16457]: * Task group "web" has 0/1 healthy allocations
May 18 08:54:06 nomad01 nomad[16457]: "
May 18 08:54:06 nomad01 nomad[16457]: 2020-05-18T08:54:00.161Z [ERROR] nomad.fsm: UpsertDeploymentPromotion failed: error="1 error occurred:
May 18 08:54:06 nomad01 nomad[16457]: * Task group "web" has 0/1 healthy allocations
May 18 08:54:06 nomad01 nomad[16457]: "
May 18 08:54:06 nomad01 nomad[16457]: 2020-05-18T08:54:00.168Z [ERROR] nomad.fsm: UpsertDeploymentPromotion failed: error="1 error occurred:
May 18 08:54:06 nomad01 nomad[16457]: * Task group "web" has 0/1 healthy allocations
May 18 08:54:06 nomad01 nomad[16457]: "
May 18 08:54:06 nomad01 nomad[16457]: 2020-05-18T08:54:00.168Z [ERROR] nomad.fsm: UpsertDeploymentPromotion failed: error="1 error occurred:
May 18 08:54:06 nomad01 nomad[16457]: * Task group "web" has 0/1 healthy allocations
May 18 08:54:06 nomad01 nomad[16457]: "
```
`job status` on the other hand says that everything is fine:
```
ID = infra.netbox
Name = infra.netbox
Submit Date = 2020-05-16T19:25:28+02:00
Type = service
Priority = 50
Datacenters = dc1
Namespace = default
Status = running
Periodic = false
Parameterized = false
Summary
Task Group Queued Starting Running Failed Complete Lost
netbox-worker 0 0 0 4 5 0
redis 0 0 0 1 4 0
web 0 0 1 2 14 0
worker 0 0 1 0 1 0
Latest Deployment
ID = 3fcc23cf
Status = successful
Description = Deployment completed successfully
Deployed
Task Group Promoted Desired Canaries Placed Healthy Unhealthy Progress Deadline
web true 1 1 1 1 0 2020-05-16T19:35:58+02:00
worker N/A 1 0 1 1 0 2020-05-16T19:35:43+02:00
Allocations
ID Node ID Task Group Version Desired Status Created Modified
c1517db2 a8efb906 web 21 run running 30m44s ago 30m10s ago
f16d9721 a8efb906 worker 21 run running 30m44s ago 30m23s ago
4ded5234 42eb20db web 21 stop complete 1d16h ago 29m48s ago
7e94f1c5 42eb20db worker 21 stop complete 1d21h ago 29m48s ago
```
It is true that a few days ago deployments did fail, so I imagine that nomad is not cleaning up those properly. I have no idea how to find them and "fix" them.
Contributor guide
No contributing guide indexed for this repository
Research direction
No source files or tests are identified in the report. Start by reproducing a Nomad restart after a failed deployment, then compare the FSM error logs with `job status`; done means the restart no longer reports erroneous promotion failures when the deployment is already successful.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100