Automattic / Automattic/hostmgr
Support VM name aliases
- Dominant language
- Swift
- Stars
- 12
- Forks
- 3
- Avg merge
- 4h 50m
- Merged PRs (30d)
- 4
Description
## Why?
It happens from time to time that the final Xcode version is just the exact same as the last RC.
For example, `xcode-16.3` was exactly the same as `xcode-16.3-rc-2`
Right now to support that use case, [we duplicate the `.vmtemplate` bundle in S3 with the new name](https://github.com/Automattic/buildkite-ci/blob/d579e6b068bc8c6a72b339cbbed37f95e14fdb03/src/agents/macos-vms/README.md#L109-L115), so that both `xcode-16.3-rc-2.vmtemplate.aar` and `xcode-16.3.vmtemplate.aar` are present in S3 and thus found by `hostmgr vm fetch` / `hostmgr vm start`.
But this obviously leads to waste of space in the S3 bucket, but also extra S3 data transfers and Mac hosts' disk space (because if a Mac host already had `xcode-16.3-rc-2` from a previous job that requested it, and a new job requires `IMAGE_ID: xcode-16.3`, it will download it all over again instead of realizing they're the same and it could reuse the RC2 it already has locally).
## How
Implement some mechanism to support alias names for VMs.
Some random ideas:
- We could have a text file named `xcode-16.3.alias`, that would contain the single line `xcode-16.3-rc-2`, and that we'd upload to the S3 bucket instead of the whole duplication of the `vmtemplate.aar`.
- Then when `hostmgr vm fetch xcode-16.3` tries to fetch the `xcode-16.3.vmtemplate.aar` file from S3 and fails to find it, instead of erroring right away it'd then check for `xcode-16.3.alias` file as a fallback, and if it exists use its content as the new VM to fetch…
- Then it'd have to remember it in some local config file so that `hostmgr vm start xcode-16.3` would know to use the `xcode-16.3-rc-2.vmtemplate` as a template to start the VM
- Or have a file in S3 named `aliases.yml` that contains the map of all VM name aliases.
- Then both `hostmgr vm fetch` and `hostmgr vm start` will check that local file/mapping first, to do any potential substitution of the IMAGE_ID alias to the VM real name to fetch/start, then they'd do the same business as usual (but with the real VM name)
- As for keeping that `aliases.yml` file in sync on Mac hosts, we could:
- Either have a cron job fetching it from S3 on a regular basis (meh)
- Or have `hostmgr` fetch it from S3 if the `hostmgr vm fetch` failed to find the VM. Meaning `hostmgr vm fetch` would do (1) check in local `aliases.yml` if the requested `IMAGE_ID` has an alias defined in this mapping, and if so use the resolved name (2) do the fetch as usual (3) if the fetch fails, download the `aliases.yml` from S3 (just in case the local one was not up-to-date), and recheck if there is a new mapping, and if so try the fetch again one last time with that new mapped name
- Or have `hostmgr vm fetch` fetch the file from S3 each time before it tries to fetch a VM… but with a `ETag`/`If-Modified-Since` request that only fetches it if it's more recent than the local one
## Additional use case
In addition to the typical case of avoiding to duplicate VM templates when having a final Xcode version be the same as the last RC, another use case for this aliasing system would be for when we end up building multiple versions of a VMs for the same Xcode because we messed up along the way (e.g. `xcode-16.0-v7`/`xcode-16.0-v9`/`xcode-16.0-v10`).
For such cases, we could define the alias `xcode-16.0` to be the latest version of those VMs, and then if a job requests `IMAGE_ID: xcode-16.0` it'd automatically point to the latest version of that VM.
Currently for those cases we can't just rename the latest VM in S3 to just `xcode-16.0` to achieve such a behavior, because that'd mean that Mac hosts that might have already downloaded a previous version of the VM would (eg. the `v7` of the VM) would not realize that they should download the new version if both the old and new version were named `xcode-16.0` without a `v` suffix (it's not like we validate the checksum of the VM is still valid every time we `vm fetch`/`vm start` to check if the same `IMAGE_ID` name still points to the exact same VM as before). But with the aliasing system, a job having `IMAGE_ID: xcode-16.0` would still first use the alias mapping to substitute the real name of the VM to fetch/start to be `xcode-16.0-v10`, and then the rest of the process would be unchanged, thus still downloading the latest VM if that alias points to a new VM now (even if it previously pointed to `xcode-16.0-v7` before and we do have that v7 VM locally)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the `hostmgr vm fetch` and `hostmgr vm start` entry points, then read the macOS VM workflow in `src/agents/macos-vms/README.md`. Decide on an alias representation and resolution flow that works with the S3 templates and local VM state. Done means alias names resolve to the intended real VM for both commands without duplicating templates or unnecessarily downloading an already available VM.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, swift
- Domain
- cli, cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100