matrix-org / matrix-org/complement
Support HS upgrade tests
- Dominant language
- Go
- Stars
- 99
- Forks
- 72
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 8
Description
Dendrite has https://github.com/matrix-org/dendrite/tree/master/cmd/dendrite-upgrade-tests which will automatically pull and run different versions of dendrite with the same database and ensure db migrations work correctly. It can be run for example by doing `./dendrite-upgrade-tests --from 0.1.0 --to 0.3.1`. This will:
- Fetch semver looking tags from github (or `HEAD`).
- Sort them.
- Pull archives for each release.
- Execute a Dockerfile with the build context set to the root directory of the release.
- Run the lowest semver, make some users and rooms, send a few messages, tear down the server.
- Keep the database as a docker volume.
- Repeat with the next lowest semver up until `--to`.
- Test that the last HS run has the original users/rooms/messages from all previous runs.
It looks like Synapse has some bisectability baked into the Complement Synapse image: https://github.com/matrix-org/complement/blob/76db9e5ae6c45d889000feb19f024f184c8bd6e3/dockerfiles/Synapse.Dockerfile#L14
If Complement standardised the format of this (e.g thou shalt accept the build arg `HS_VERSION=$semver`) then we could, in theory, make https://github.com/matrix-org/dendrite/tree/master/cmd/dendrite-upgrade-tests work for any HS (modulo working out how to do persistence, as currently Dendrite uses postgres and that assumption is baked into the upgrade testing infra: this could just be stating that the directory `/data` is persisted across runs and you need to just dump your DB there).
This arguably is and isn't feature creep. It is feature creep because it has nothing to do with the integration tests in `/tests`. It isn't feature creep because it is still ultimately testing HSes, just the upgrade paths.
Thoughts @richvdh (on the basis that you added `SYNAPSE_VERSION`?
Contributor guide
Research direction
Start by reading cmd/dendrite-upgrade-tests and Complement's /tests infrastructure, then inspect dockerfiles/Synapse.Dockerfile where SYNAPSE_VERSION is handled. Clarify the proposed HS_VERSION and persistence contract before implementation; done should include a documented, repeatable upgrade test path for supported homeservers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go, postgres
- Domain
- databases, devops, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100