cockroachdb / cockroachdb/cockroach

roachprod: prebuild roachprod/roachtest cloud images

Open
#150,144 1 comment 0 reactions 1 assignee Claimed by @golgeek View on GitHub
A-testeng-infra A-testing C-enhancement T-testeng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

### Problem Statement

Provisioning roachprod clusters has become increasingly slow because we configure each VM entirely at boot time via the cloud-init startup script:

* We install a growing list of Debian packages and test-only tools (e.g. observability exporters).
* Instance startup time keeps creeping us as the package list grows.
* Every boot depends on live external endpoints (Debian mirrors, GitHub tarballs, etc.). A transient outage or rate-limit could fail roachtests as infra flake.
* While dependency versions are pined, downloading tarballs or building external dependencies for each test could mean that the same test run is not perfectly reproducible over time.

See [cockroachdb/cockroach#147352](https://github.com/cockroachdb/cockroach/issues/147352) for the ongoing effort to package several of these tools as first-class `.deb` artifacts instead of downloading or compiling from source at boot.

---

### Proposal — Pre-baked Images with Packer + Ansible

1. Image build phase (offline)
1. For each cloud provider (GCE, AWS, Azure, …) and each architecture we support (amd64, arm64, fips, s390x), use Hashicorp Packer to spin up a temporary VM.
2. Apply an Ansible role that:
* Upgrades the OS.
* Installs all baseline OS packages and our test dependencies.
* Adds the pre-built `.deb` packages (created in #147352).
* Copies any custom binaries we cannot (yet) package.
3. Output an image/AMI ready for test workloads.

2. Test run phase (online)
* `roachprod` launches from the pre-baked image instead of a vanilla Ubuntu image.
* `cloud-init` now only needs to:
* Attach extra disks.
* Write CockroachDB configuration and cluster topology.
* Start the node.

> Note
> The software discussed in #147352 originally built ready-to-go AMIs via Packer in GCP Cloud Build behind a small REST API. Re-implementing that workflow for Test Engineering should be straightforward.

---

### Benefits

* Boot time: instances start significantly faster because all heavy installation happens once, offline.
* Stability: test runs no longer depend on third-party package mirrors or GitHub availability.
* Reproducibility: the exact image is versioned and immutable; restoring an old CI run is as simple as pinning an AMI ID.

---

### Drawbacks / Trade-offs

| Drawback | Mitigation |
| -------- | ---------- |
| Images must be rebuilt whenever any dependency changes. | Automate nightly image builds in CI; only promote to “latest” when the Ansible role finishes successfully. |
| More images to manage (per cloud, per arch, per Ubuntu release). | Store metadata (build date, upstream Ubuntu SHA) in image tags to aid pruning; use Terraform data sources to fetch “latest-stable”. |
| Slightly higher storage cost for custom images. | Negligible. |

---

### Acceptance Criteria

- [ ] Packer templates exist for each supported cloud / arch.
- [ ] Ansible role reproduces current cloud-init behavior minus CRDB setup.
- [ ] CI job builds an image on demand and publishes its ID/URI.
- [ ] `roachprod` can consume a `--image-family` flag (or similar) to use the new images.

Jira issue: CRDB-52590

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.