hashicorp / hashicorp/nomad

Fail deployments if promotion returns error after deadline

Open
#6,740 3 comments 0 reactions 0 assignees View on GitHub
theme/deployments type/enhancement
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

Follow-up improvement coming out of https://github.com/hashicorp/nomad/issues/6407#issuecomment-555519437

For each allocation that's part of a deployment, we start up a health hook on the client. Once that hook updates the deployment status of the alloc to healthy (or the deadline passes without doing so), the hook exits. If the allocation fails after it initially reports it's healthy, we're in a state where the deployment thinks all its placements are healthy, but when we promote the deployment to make that a reality, the server's state machine rejects the change because there aren't enough canary allocations that are actually healthy.

This is a pathological case which we're only likely to hit when promotions are made manually or when tasks are slow to start _and_ flappy after start, and getting the behavior to be predictable and understandable to operators in that condition is difficult. There may be some future improvements we can make to the deployments (especially with L7 health checks via Connect on the horizon).

In the meantime, we're going to make a change so that promotions are marked as failed if the promotion hits an error past the promotion deadline.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing Nomad's deployment promotion flow and the client health-hook behavior described in the issue, then inspect the server state-machine handling of promotion errors and deadlines. Done means a promotion error occurring after the promotion deadline marks the deployment as failed, with behavior covered by the relevant existing tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.