bottlerocket-os / bottlerocket-os/bottlerocket-update-operator

Consider adding a configurable auto-rollback mechanism

Open
#590 0 comments 0 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
Rust
Stars
220
Forks
50
PR merge metrics
No merged PRs in 30d

Description

**Issue or Feature Request:**

To perform a rollback, you would need to use Bottlerocket’s settings API to [lock a node to a given version](https://bottlerocket.dev/en/os/1.17.x/api/settings/updates/#version-lock). Brupop does some work to prevent errant updates from rolling out across the whole cluster: if the Brupop agent fails to come up on an updated node, then that node will never re-enter the “Idle” state from Brupop’s perspective. Since only a finite (configurable) number of nodes can be updated at a time, this means that complete update failures have a configurable maximum number of nodes that can be impacted by a Brupop-managed update.

Unfortunately, this will not help in cases where updates don’t affect the node’s ability to boot or Brupop’s ability to successfully manage the node, but do impair other applications. One helpful feature is that Brupop respects PodDisruptionBudgets, so using PDBs on an application can prevent Brupop from continuing to drain more hosts while older ones are impaired.

It may be helpful to consider a feature that allows Brupop to rollback nodes under some configurable circumstance. In practice, this would only work in cases where the Brupop agent itself comes back up but applications do not. We would need some way for users to “hook” their application health into Brupop.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.