compiler-explorer / compiler-explorer/ce-ci

Commit the lambda zips and drive the apply from a Makefile, instead of a second root module

Open
#27 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
HCL
Stars
2
Forks
2
PR merge metrics
No merged PRs in 30d

Description

> Written by Claude (an LLM), at Matt's request, following the stale-lambda incident on 2026-08-14.

## Problem

The lambda zips are fetched out of band into a gitignored path, and the module version is declared in two places that can silently diverge. On 2026-08-14 we found the deployed lambdas were years older than the pinned `v7.9.0`: someone had bumped the version but not re-run `cd lambdas-download && terraform apply`, so every apply since had redeployed ancient lambda code alongside new configuration. Nothing detected it, and it only surfaced because an unrelated change made someone read a plan.

## Why the current shape exists (do not "fix" it by inlining)

The split root module is a workaround for a real terraform constraint, not an accident. Upstream hashes the zip at **plan** time (`modules/lambda/main.tf:24` at v7.9.0):

```hcl
source_code_hash = var.lambda.s3_bucket == null ? filebase64sha256(var.lambda.zip) : null
```

`filesha256()` on a file created by an apply-time `local-exec` is a hard plan error, so a `null_resource` download inside the main root cannot work. Terraform also has no root-to-root apply cascade (`terraform_remote_state` reads, never applies). And a module `version` argument must be a literal, so it cannot be shared as a variable.

## Proposal

Commit the zips, and make the apply depend on them, copying the idiom already used in `compiler-explorer/infra` (`infra/Makefile:186-191` makes `terraform-apply` depend on `upload-lambda`, so the artifact step cannot be skipped).

- [ ] Commit the four zips to `lambdas/*.zip` (~2.4 MB total) and point `main.tf:75-78` at `${path.module}/lambdas/*.zip`. This also fixes a latent bug: those paths are currently relative to the process CWD, not the module.
- [ ] Add `update-lambdas.sh `: curl the release assets, verify each against the `digest` field on the GitHub release, fail loudly on mismatch.
- [ ] Add a `Makefile` whose `plan`/`apply` targets depend on a `lambdas` target that verifies the committed zips and re-downloads only on mismatch, deriving the tag from the single literal in `main.tf`.
- [ ] Delete `lambdas-download/` and its separate state.
- [ ] Update `README.md` and `.claude/commands/upgrade.md` (the latter has stale line numbers and tells you to commit a gitignored lock file).

Result: **one version declaration, one apply, and a check that blocks rather than warns.**

## Alternatives considered

- **Version-addressed S3 keys** (`ce-ci/lambdas/v7.9.0/webhook.zip`, read via `data.aws_s3_object` as `infra/terraform/lambda.tf:115-125` does). The strong runner-up: keeps binaries out of git and stays immutable, at the cost of an upload step. Note the module sets `source_code_hash = null` on the S3 path, so change detection has to come from the key or object versioning. Prefer this if committing binaries is unacceptable.
- **Keep the split and add a plan-time digest check** (#26). Detects a mistake the other shapes make impossible, warns rather than blocks, and adds a third version literal. Closed in favour of this issue; its digest logic should be salvaged into `update-lambdas.sh`.
- **Git LFS for the zips.** Actively harmful here: an unfetched pointer would have terraform hash a 130-byte text file and deploy that as a lambda.

On repo growth: there have been exactly two version bumps in this repo's history, and the pack is currently 29 KB. At 2-3 bumps a year this is under 10 MB per decade. The decisive property is not size but that the git tree would then fully determine what is deployed, so a bump that forgot the artifacts is visible in review as "version changed, binaries unchanged".

## Related, found while investigating

- `runner_version = "2.335.1"` is declared three times (`packer-vars.hcl:13`, `packer-vars-arm64.hcl:11`, `packer-vars-win-builder.hcl:5`) and can silently diverge; per the README a stale one wedges jobs. `infra/packer/win.pkr.hcl:34` shows the house fix: one file consumed by many, via `trimspace(file(...))`.
- `lambdas-download/main.tf`'s `module "lambdas"` has a `source` but no `version`, so that module is unpinned.
- This repo has no CI (no `.github/`), which is why none of the above is caught mechanically.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading main.tf:75-78, modules/lambda/main.tf:24 at v7.9.0, and the compiler-explorer/infra Makefile:186-191 to understand the plan-time hash constraint and dependency pattern. Review the existing lambdas-download state, digest logic from issue #26, README.md, and .claude/commands/upgrade.md; done means committed verified zips, a working update script and Makefile-driven plan/apply, removal of the second root, and updated documentation.

Written by the indexing model from the issue text.

Assessment

Tech stack
shell, terraform
Domain
build-system, devops, documentation, infrastructure
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.