bottlerocket-os / bottlerocket-os/bottlerocket-update-operator

Upgrade managed node groups AMI ID without restarting nodes

Open
#705 1 comment 2 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
220
Forks
50
PR merge metrics
No merged PRs in 30d

Description

When using managed node groups with Bottlerocket and Brupop, we are facing a situation of nodes starting with an older version of Bottlerocket.

As stated in the docs (https://bottlerocket.dev/en/brupop/1.3.x/troubleshoot/#bottlerocket-instances-start-with-an-old-version-of-bottlerocket), this is due to EKS's managed node groups having an AMI ID fixed to an older version.

The fix for this could be to upgrade EKS's managed node group version.
However, this lead to a slow operation (30 minutes - 1 hour) on EKS (see https://github.com/aws/containers-roadmap/issues/1619)

As nodes are already upgraded by Brupop, this operation is useless.

In order to avoid the slow upgrade operation on EKS, we would like the managed node groups to create new nodes in managed node group with the same Bottlerocket version as the one updated by Brupop.

However, this doesn't seems possible at the moment, every AMI ID change in managed node groups will rollout on all nodes.

### Workaround

In this issue (https://github.com/bottlerocket-os/bottlerocket-update-operator/issues/45), we've seen people working around this by using a custom node group (not managed) with SSM parameters.

However, the impact on this is that node management is not handled with the Kubernetes API with the flow that cordon, drain etc...
Because of this, Pod Disruption Budgets are not respected, which can lead to outages.

So, I'd like to know : what is the recommended way to use brupop with managed node groups? Is there a way to avoid waiting for managed node groups rollout?

**Image I'm using:** public.ecr.aws/bottlerocket/bottlerocket-update-operator:v1.4.0

Contributor guide

Open the contributing guide

Research direction

No implementation files, tests, or entry points are named. Start with the linked Brupop troubleshooting documentation and the EKS managed node group rollout issue, then inspect how managed node groups are handled; done means documenting or implementing a supported workflow that avoids an unnecessary rollout while preserving Kubernetes node-management behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes
Domain
cloud, devops, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.