Fail deployments if promotion returns error after deadline
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
Follow-up improvement coming out of https://github.com/hashicorp/nomad/issues/6407#issuecomment-555519437
For each allocation that's part of a deployment, we start up a health hook on the client. Once that hook updates the deployment status of the alloc to healthy (or the deadline passes without doing so), the hook exits. If the allocation fails after it initially reports it's healthy, we're in a state where the deployment thinks all its placements are healthy, but when we promote the deployment to make that a reality, the server's state machine rejects the change because there aren't enough canary allocations that are actually healthy.
This is a pathological case which we're only likely to hit when promotions are made manually or when tasks are slow to start _and_ flappy after start, and getting the behavior to be predictable and understandable to operators in that condition is difficult. There may be some future improvements we can make to the deployments (especially with L7 health checks via Connect on the horizon).
In the meantime, we're going to make a change so that promotions are marked as failed if the promotion hits an error past the promotion deadline.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing Nomad's deployment promotion flow and the client health-hook behavior described in the issue, then inspect the server state-machine handling of promotion errors and deadlines. Done means a promotion error occurring after the promotion deadline marks the deployment as failed, with behavior covered by the relevant existing tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100