cowprotocol / cowprotocol/services
chore: parallelize e2e tests
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 318
- Forks
- 189
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 121
Description
Background
Currently our e2e test suite is quite slow to run because every test has to run serially. The reason is that those tests rely on a DB and anvil instance and we currently only spawn 1 of each for all tests to share. So in order to not interfere with each other the tests need to run serially.
A common workflow is to implement a feature, run the fast tests locally (unit tests, DB tests) and then push to a branch so that CI runs the e2e tests for you. This is clearly a sub-optimal feedback loop if you only find out that your change broke something 15m after you already switched to something else. So the goal would be to make e2e tests fast enough to run locally such that you don't have to switch between work streams all the time. This will also be nice for agentic workflows which profit immensely from faster verifications.
Details
I think there are mainly 2 approaches: 1) dockerize every relevant component (DB, anvil) and have 1 instance per test, 2) keep things on the host system as much as possible (e.g. 1 separate DB per postgres instance, 1 anvil process per test).
While option 2 would probably involve less work and might be slightly more performant I think orchestrating an isolated docker environment per test is the way to go.
This 1) does not put any requirements on the host system, 2) is OS agnostic and 3) is more future proof in case our e2e tests start to rely on more machinery.
While the docker orchestration will be the main part of the work we also hardcode ports in a few places which will have to be made generic.
Making those hardcoded things dynamic can easily be fixed and merged before even touching the docker orchestration.
Acceptance criteria
an arbitrary number of e2e tests can run in parallel without affecting each other.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the e2e test harness, the shared database and Anvil setup, and the places where ports are hardcoded. Compare the proposed host-based and Docker-isolated approaches, then verify that the chosen orchestration supports concurrent runs without interference. Done means an arbitrary number of e2e tests can run in parallel successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, rust
- Domain
- devops, infrastructure, testing-qa
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100