hashicorp / hashicorp/nomad

Nomad restart logs Error about previously failed canary deployment

Open
#7,997 0 comments 0 reactions 0 assignees View on GitHub
stage/needs-investigation theme/deployments
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
Nomad v0.11.2 (807cfebe90d56f9e5beec3e72936ebe86acc8ce3)

### Operating system and Environment details
Debian Stable

### Issue
When restarting nomad I get the following error in the logs:
```
May 18 08:54:06 nomad01 nomad[16457]: 2020-05-18T08:54:00.161Z [ERROR] nomad.fsm: UpsertDeploymentPromotion failed: error="1 error occurred:
May 18 08:54:06 nomad01 nomad[16457]: * Task group "web" has 0/1 healthy allocations
May 18 08:54:06 nomad01 nomad[16457]: "
May 18 08:54:06 nomad01 nomad[16457]: 2020-05-18T08:54:00.161Z [ERROR] nomad.fsm: UpsertDeploymentPromotion failed: error="1 error occurred:
May 18 08:54:06 nomad01 nomad[16457]: * Task group "web" has 0/1 healthy allocations
May 18 08:54:06 nomad01 nomad[16457]: "
May 18 08:54:06 nomad01 nomad[16457]: 2020-05-18T08:54:00.168Z [ERROR] nomad.fsm: UpsertDeploymentPromotion failed: error="1 error occurred:
May 18 08:54:06 nomad01 nomad[16457]: * Task group "web" has 0/1 healthy allocations
May 18 08:54:06 nomad01 nomad[16457]: "
May 18 08:54:06 nomad01 nomad[16457]: 2020-05-18T08:54:00.168Z [ERROR] nomad.fsm: UpsertDeploymentPromotion failed: error="1 error occurred:
May 18 08:54:06 nomad01 nomad[16457]: * Task group "web" has 0/1 healthy allocations
May 18 08:54:06 nomad01 nomad[16457]: "
```
`job status` on the other hand says that everything is fine:
```
ID = infra.netbox
Name = infra.netbox
Submit Date = 2020-05-16T19:25:28+02:00
Type = service
Priority = 50
Datacenters = dc1
Namespace = default
Status = running
Periodic = false
Parameterized = false

Summary
Task Group Queued Starting Running Failed Complete Lost
netbox-worker 0 0 0 4 5 0
redis 0 0 0 1 4 0
web 0 0 1 2 14 0
worker 0 0 1 0 1 0

Latest Deployment
ID = 3fcc23cf
Status = successful
Description = Deployment completed successfully

Deployed
Task Group Promoted Desired Canaries Placed Healthy Unhealthy Progress Deadline
web true 1 1 1 1 0 2020-05-16T19:35:58+02:00
worker N/A 1 0 1 1 0 2020-05-16T19:35:43+02:00

Allocations
ID Node ID Task Group Version Desired Status Created Modified
c1517db2 a8efb906 web 21 run running 30m44s ago 30m10s ago
f16d9721 a8efb906 worker 21 run running 30m44s ago 30m23s ago
4ded5234 42eb20db web 21 stop complete 1d16h ago 29m48s ago
7e94f1c5 42eb20db worker 21 stop complete 1d21h ago 29m48s ago
```

It is true that a few days ago deployments did fail, so I imagine that nomad is not cleaning up those properly. I have no idea how to find them and "fix" them.

Contributor guide

No contributing guide indexed for this repository

Research direction

No source files or tests are identified in the report. Start by reproducing a Nomad restart after a failed deployment, then compare the FSM error logs with `job status`; done means the restart no longer reports erroneous promotion failures when the deployment is already successful.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.