element-hq / element-hq/synapse

getting rid of the replication "stream" tables

Open
#13,456 3 comments 0 reactions 0 assignees View on GitHub
A-Database A-Workers T-Task Z-Cleanup
Dominant language
Python
Stars
4.6k
Forks
600
Avg merge
5d 22h
Merged PRs (30d)
51

Description

This issue has been migrated from [#13456](https://github.com/matrix-org/synapse/issues/13456).

---

Currently we have a number of tables in the database that exist only to record data for the replication streams. These include:
* `cache_invalidation_stream_by_instance`
* `current_state_delta_stream`
* `ex_outlier_stream`
* `presence_stream`
* `push_rules_stream`

(and there may well be others).

Essentially, whenever we record a *change* to a table that needs to be replicated to workers, we add a row to one of these tables; the rows are then used in one of two ways:
* `ReplicationStreamer._run_notifier_loop` regularly polls them (via the `*Stream._update_function` methods) and sends out a `NOTIFY` over [Redis pubsub](https://redis.io/docs/manual/pubsub/) with the data from the table.
* If a worker gets disconnected from Redis (so misses notifications), it can catch up with any missed notification by reading the relevant table itself.

The reason we use this arrangement is twofold:
1. It allows workers which miss the memo (because they were disconnected from Redis) to catch up with anything they missed.
2. Since the Redis notifications are sent out asynchronously by `_run_notifier_loop`, it is possible for the "writing" process to abort between updating the database and sending the notification to Redis. Persisting the data in postgres ensures that we can replay anything that wasn't sent when we restart.

However, I assert that these extra tables are a source of complexity, as well as increased database I/O and storage (not least because we never clear them out (#5888)). Worse, whenever we need to add a new type of replication stream, we have to add a load of extra paraphenalia in the shape of a new stream table. It would be good to consider how to get rid of them.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.