multiregion rollbacks result in all "blocked"
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
If a multiregion deployment fails, the deploymentwatcher marks other regions as failed depending on the `on_failure` setting. The rollback in a given region is controlled by the `auto_revert` field on `update`. When a multiregion deployment with `multiregion.strategy.on_failure="all"` and `update.auto_revert=true` fails, all regions get failed and rolled back. Rolling back creates a new deployment for the rollback in each region. These rollback deployments never exit the `blocked` state.
This is arguably an improvement over the existing single-region behavior where:
- `nomad deployment list` shows a failed deployment and then a successful deployment (the rollback)
- `nomad job status` will show a successful deployment, because it shows the _latest_ deployment, which is the rollback deployment.
With multiregion deployments we get:
- `nomad deployment list` shows a failed deployment and then a successful deployment (the rollback)
- `nomad job status` will show a blocked deployment.
The operator can unblock these deployments manually or simply leave it in place because all the allocations have been placed. However, it's not very consistent and might be confusing to operators.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the deploymentwatcher path for multiregion failures and the creation of rollback deployments when update.auto_revert is enabled. Done means rollback deployments no longer remain blocked and nomad job status reports the resulting deployment consistently with the single-region behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100