hashicorp / hashicorp/nomad

multiregion rollbacks result in all "blocked"

Open
#8,210 0 comments 2 reactions 0 assignees View on GitHub
theme/bad-ux theme/deployments
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

If a multiregion deployment fails, the deploymentwatcher marks other regions as failed depending on the `on_failure` setting. The rollback in a given region is controlled by the `auto_revert` field on `update`. When a multiregion deployment with `multiregion.strategy.on_failure="all"` and `update.auto_revert=true` fails, all regions get failed and rolled back. Rolling back creates a new deployment for the rollback in each region. These rollback deployments never exit the `blocked` state.

This is arguably an improvement over the existing single-region behavior where:
- `nomad deployment list` shows a failed deployment and then a successful deployment (the rollback)
- `nomad job status` will show a successful deployment, because it shows the _latest_ deployment, which is the rollback deployment.

With multiregion deployments we get:
- `nomad deployment list` shows a failed deployment and then a successful deployment (the rollback)
- `nomad job status` will show a blocked deployment.

The operator can unblock these deployments manually or simply leave it in place because all the allocations have been placed. However, it's not very consistent and might be confusing to operators.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the deploymentwatcher path for multiregion failures and the creation of rollback deployments when update.auto_revert is enabled. Done means rollback deployments no longer remain blocked and nomad job status reports the resulting deployment consistently with the single-region behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.