letsencrypt / letsencrypt/boulder
Revamp how we run tests
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 5.8k
- Forks
- 649
- Avg merge
- 3d 23h
- Merged PRs (30d)
- 24
Description
Objective: make all Boulder tests runnable just by running go test ./....
Obstacle 1: Some of the RA tests are backed by insufficient fakes, and instead rely on an in-memory SA connected to a real database instance running in a container.
Obstacle 2: The integration tests require the boulder-tools container to have access to python (for the integration test harness), rust (for spellcheck), protoc-gen-go, and more.
Obstacle 3: The integration tests require lots of running boulder binaries, as well as support software like redis, mysql, and the things representing "the internet" (s3-test-srv, challtestsrv, etc).
Proposed solution:
- We produce two container images: "boulder" and "boulder-tools".
- The "boulder" container contains only the boulder binary. It is the release artifact. It is also used by our integration tests (details below).
- The "boulder-tools" container holds all our support infrastructure, like challtestsrv, protoc-gen-go, and golangci-lint.
- The RA unit tests which rely on a real database are split into a separate file and updated to use testcontainers to spin up the necessary dependency container.
- The integration tests are updated in a similar way:
- The //test/integration folder gets a TestMain function which spins up all the necessary containers (maybe using testcontainers' compose support)
- Every boulder service gets its own "boulder" container (or more than one, for redundant services)
- The public internet is represented by one or more "boulder-tools" containers
- The integration tests can make network requests to the WFE container, SQL requests to the database container, CAA mocking requests to challtestsrv in the tools container, etc.
Optionally, we can still retain the t.sh wrapper, to simplify things like setting the BOULDER_CONFIG_DIR environment variable and specifying the -tags=integration build tag, for when you only want to run the lighter/faster unit tests.
This approach eliminates the need for tests to be run from a container; the dev can run a simple 'go test' and expect it to work. This approach eliminates the need for startservers.py. It even eliminates the need for the root of our repo to be cluttered with test-only files like docker-compose.yml. This approach also leans in to using the boulder production release artifact as the test artifact.
I'm sure there are lots of cons to this approach, and alternative approaches that are even better. Let's use this bug to brainstorm the best architecture here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the //test/integration folder and the current t.sh, startservers.py, and docker-compose.yml setup. Map which containers and external services the existing tests require, then compare that with the proposed boulder and boulder-tools split. Done means the stated test suites can run with go test ./..., with any remaining integration-test invocation documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go, mysql, python, redis, rust
- Domain
- devops, infrastructure, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100