element-hq / element-hq/synapse

Complement `TestOIDCProviderUnavailable` should use a clean deployment (cross-test pollution from modified homeserver config being re-used)

Open
#19,937 0 comments 1 reaction 0 assignees View on GitHub
A-Testing
Dominant language
Python
Stars
4.6k
Forks
600
Avg merge
5d 22h
Merged PRs (30d)
51

Description

We enable [`COMPLEMENT_ENABLE_DIRTY_RUNS`](https://github.com/element-hq/synapse/blob/c63d77a79d7157f26f849684520ba9e99f4d07c0/scripts-dev/complement.sh#L309-L311) ([docs](https://github.com/matrix-org/complement/blob/0e6f8552ff0c99fddb97222399efed3e1f0cb91a/ENVIRONMENT.md#complement_enable_dirty_runs)) which means Complement will reuse deployments (shares homeservers between tests).

Because we modify the homeserver configuration in `TestOIDCProviderUnavailable`, this also affects all other tests that run after this test.

https://github.com/element-hq/synapse/blob/c63d77a79d7157f26f849684520ba9e99f4d07c0/complement/tests/oidc_test.go#L65-L72

### Potential solutions

As suggested in the [`COMPLEMENT_ENABLE_DIRTY_RUNS` docs](https://github.com/matrix-org/complement/blob/0e6f8552ff0c99fddb97222399efed3e1f0cb91a/ENVIRONMENT.md#complement_enable_dirty_runs), we could use `OldDeploy` to get a fresh deployment. The problem is that the `OldDeploy` API is way more cumbersome using blueprints instead of just being able to specify the number of homeservers desired.

Ideally, we'd just be able to specify that this homeserver should not be re-used when making the deployment like `deployment := complement.CleanDeploy(t, 1)` (perhaps more apparent name)

Contributor guide

Open the contributing guide

Research direction

Inspect complement/tests/oidc_test.go at TestOIDCProviderUnavailable and the COMPLEMENT_ENABLE_DIRTY_RUNS documentation; compare the existing OldDeploy API with the proposed clean deployment entry point. Done means this test receives a non-reused homeserver and later Complement tests no longer inherit its modified configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.