element-hq / element-hq/synapse

Gradual migration steps for Sliding Sync tables (`sliding_sync_joined_rooms`, `sliding_sync_membership_snapshots`)

Open
#17,623 0 comments 0 reactions 0 assignees View on GitHub
A-Database A-Sync
Dominant language
Python
Stars
4.6k
Forks
600
Avg merge
5d 22h
Merged PRs (30d)
51

Description

This issue is meant to track the [gradual migration](https://github.com/element-hq/synapse/blob/e75a23a63d5fe8edb0145bdd55b98d9e6a29b2ab/docs/development/database_schema.md#worked-examples-of-gradual-migrations) so that the application code can fully rely on the `sliding_sync_joined_rooms`/`sliding_sync_membership_snapshots` database tables.

(an example of gradually migrating to a completely new database table)

### Synapse version `N + 1`

*This step was done in https://github.com/element-hq/synapse/pull/17512 (landed in Synapse vTODO)*

The `N+1` step introduces the `sliding_sync_joined_rooms`/`sliding_sync_membership_snapshots` tables and background updates to populate those tables over time. `SCHEMA_VERSION` was bumped to `87`.

Since we always make sure to fully insert a row if it doesn't exist, if a row exists in those tables, it can be relied upon. But missing rows need to use a fallback to get the same data.

### Synapse version `N + 2`

In reality, this might happen more than one Synapse version down the line to allow more time for people to upgrade and run the background update.

Bump `SCHEMA_VERSION` to `88` and bump `SCHEMA_COMPAT_VERSION` to at-least `87` because we don't want people to downgrade and miss writes while they are on an older version. Add a foreground update to finish off populating the tables completely so we can read from new tables without the fallback. Application code can now rely on the new tables being populated.

Code pointing at this issue can also be removed in this step (search the codebase for `https://github.com/element-hq/synapse/issues/17623`)

- The `sliding_sync_joined_rooms_to_recalculate` database table can be dropped
- Remove the `_resolve_stale_data_in_sliding_sync_tables(...)` catch-up logic and the associated tests (`SlidingSyncTablesCatchUpBackgroundUpdatesTestCase`)
- Remove the fallback logic in the Sliding Sync API used to grab the data that we can now strictly get from the new database tables.

Contributor guide

Open the contributing guide

Research direction

Read the gradual-migration worked examples in docs/development/database_schema.md, then search for issue 17623 and inspect the schema-version migration and Sliding Sync API code. Use SlidingSyncTablesCatchUpBackgroundUpdatesTestCase and the named catch-up logic as the testing entry points; done means the tables are fully populated, the fallback and catch-up paths are removed, and the obsolete table is dropped.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, databases
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.