aws / aws/containers-roadmap

[EKS] [request]: Speed up cluster autoscale by automatically creating new EKS Optimized AMIs with security patches

Open
#1,712 3 comments 3 reactions 0 assignees View on GitHub
EKS Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**
Downloading and installing securty patches can add 45 seconds to the boot time when the Cluster Autoscale adds new nodes. This affect responsiveness to traffic spikes. Therefore, the request is twofold:

1) Add a process which automatically releases new sub-versions of EKS Optimized AMIs containing security patches whenever a new security patch is released that would normally be installed during boot of the original AMI.
2) Add an SSM API that receives an AMI ID and returns the ID of a new AMI that is exactly the same as the original, but has all security patches

**Which service(s) is this request for?**
EKS

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**

One of the factors that affect the responsiveness of the Cluster autoscaler is how long it takes new nodes to initialize. Part of the node bootup is checking/download/installation of security patches. If all security patches are included in the AMI, the bootup time is reduced by as much as 45 seconds.

Even though new AMIs are released approximately every 10 days, it is not good enough to just always use the latest recommended AMI for two reasons:

1) New AMI versions contain changes that are not just security related and therefore it is dangerous to automatically take them into production systems without testing (see https://github.com/aws/containers-roadmap/issues/319). However, it should be perfectly safe to take a new AMI that is identical to the original one except for the security patches, since even the original AMI will anyway download/install these same security patches during initial boot.

2) Security patches are released more often than recommended AMIs. In a recent test, I took an AMI that was 6 days old, and when the node booted, it downloaded/installed 4 security patches. When I created a custom AMI that included these 4 patches, node bootup time was reduced by 45 seconds. See attached excerpts from from cloud-init-output.log for the recommended AMI versus the custom AMI

So the request is to:
1) Create an automated procedure that will create new AMIs with a naming convention that includes the original AMI name, plus a patch version suffix. For example: amazon-eks-node-1.21-v20220406-p1, amazon-eks-node-1.21-v20220406-p2, etc

2) Add an SSM API where we can pass in the original AMI name (e.g. amazon-eks-node-1.21-v20220406) and get back the ID/Name of the latest patched AMI which is based on the original AMI. This can be used to automate updating of AutoScaling Groups with the patched AMI ID.

**Are you currently working around this issue?**
We are considering using the techniques described in https://docs.aws.amazon.com/systems-manager/latest/userguide/automation-walk-ami-patching.html to create a lambda that will periodically check for security updates, generate new custom AMIs, and patch the ASG. However, we are probably not the only ones who can benefit from this.

**Attachments**

[custom-cloud-init-output.log](https://github.com/aws/containers-roadmap/files/8487647/custom-cloud-init-output.log)
[recommended-cloud-init-output.log](https://github.com/aws/containers-roadmap/files/8487648/recommended-cloud-init-output.log)

Contributor guide

Open the contributing guide

Research direction

Start with the issue's attached cloud-init logs and the AWS Systems Manager automation-walk-ami-patching documentation. The requested scope is an automated process for patched EKS Optimized AMIs plus an SSM lookup for the latest patched AMI based on an original AMI. Done means both capabilities are specified and implemented, including the proposed naming and patch-version behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.