element-hq / element-hq/synapse
Complement `TestOIDCProviderUnavailable` should use a clean deployment (cross-test pollution from modified homeserver config being re-used)
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 600
- Avg merge
- 5d 22h
- Merged PRs (30d)
- 51
Description
We enable [`COMPLEMENT_ENABLE_DIRTY_RUNS`](https://github.com/element-hq/synapse/blob/c63d77a79d7157f26f849684520ba9e99f4d07c0/scripts-dev/complement.sh#L309-L311) ([docs](https://github.com/matrix-org/complement/blob/0e6f8552ff0c99fddb97222399efed3e1f0cb91a/ENVIRONMENT.md#complement_enable_dirty_runs)) which means Complement will reuse deployments (shares homeservers between tests).
Because we modify the homeserver configuration in `TestOIDCProviderUnavailable`, this also affects all other tests that run after this test.
https://github.com/element-hq/synapse/blob/c63d77a79d7157f26f849684520ba9e99f4d07c0/complement/tests/oidc_test.go#L65-L72
### Potential solutions
As suggested in the [`COMPLEMENT_ENABLE_DIRTY_RUNS` docs](https://github.com/matrix-org/complement/blob/0e6f8552ff0c99fddb97222399efed3e1f0cb91a/ENVIRONMENT.md#complement_enable_dirty_runs), we could use `OldDeploy` to get a fresh deployment. The problem is that the `OldDeploy` API is way more cumbersome using blueprints instead of just being able to specify the number of homeservers desired.
Ideally, we'd just be able to specify that this homeserver should not be re-used when making the deployment like `deployment := complement.CleanDeploy(t, 1)` (perhaps more apparent name)
Contributor guide
Research direction
Inspect complement/tests/oidc_test.go at TestOIDCProviderUnavailable and the COMPLEMENT_ENABLE_DIRTY_RUNS documentation; compare the existing OldDeploy API with the proposed clean deployment entry point. Done means this test receives a non-reused homeserver and later Complement tests no longer inherit its modified configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100