[CI] Automatically rotate worker machine images
- Dominant language
- C++
- Stars
- 28.8k
- Forks
- 8.9k
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 54
Description
#8142 automated the process for building the image for worker machine. (*) We now have automated pipelines that builds machine images. However, the pipelines need to be invoked manually. I might forget to run them, and the worker images would become outdated. We must automate rotation of worker machine images.
**Background**
We currently build two custom machine images:
* `linux-amd64-gpu`: Amazon Linux, AMD64 arch, comes with CUDA and NVIDIA Docker installed
* `windows-gpu`: Windows, AMD64 arch, comes with CUDA, Visual Studio and Conda installed (**)
The custom machine images are built **on top of** images provided by Buildkite:
* `linux-amd64-gpu` is built on top of `linuxamd64` (Amazon Linux 2 (64-bit x86))
* `windows-gpu` is built on top of `windows` (Amazon Linux 2 (64-bit x86))
In [the EC2 Image Builder service](https://aws.amazon.com/image-builder/), we specify two inputs to the builder pipeline:
1. ID of the base image.
2. Recipe for installing drivers and other packages.
The pipeline loads the base machine image, applies the recipe, and then publishes a new machine machine with a new AMI ID.
There are currently two steps that are performed **manually**:
* The AMI ID of the base image must be specified in `tests/buildkite/infrastructure/aws-stack-creator/metadata.py`. When Buildkite publishes new versions of `linuxamd64` and `windows`, they receive new AMI IDs, so we need to update `metadata.py`.
* We need to manually invoke the EC2 Image pipeline to obtain new worker images.
I want to get rid of these manual steps. They should be performed automatically.
**TODOs**
- [ ] Build a Lambda function to periodically check the content of https://s3.amazonaws.com/buildkite-aws-stack/latest/aws-stack.yml. Whenever Buildkite publishes new versions of the base images (`linuxamd64` and `windows`), the updated AMI IDs should be registered in [the SSM Parameter Store](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html).
- [ ] Build another Lambda function to invoke the EC2 Image Builder pipeline periodically. The base image's AMI ID should be fetched from the Parameter Store. Also, the AMI ID of the newly built custom image should be published at the Parameter Store.
- [ ] Update `tests/buildkite/infrastructure/aws-stack-creator/create_stack.py` so that it fetches all AMI IDs from the Parameter Store.
(*) Read [Amazon Machine Image (AMI)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html) to read more about what constitutes a "machine image." In short, an AMI consists of system components that are needed to launch a VM in the cloud, such as OS and drivers.
(**) Rather unfortunately, we can't use NVIDIA Docker on Windows. So all development tools need to be part of the Windows worker's machine image.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.