element-hq / element-hq/synapse
getting rid of the replication "stream" tables
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 600
- Avg merge
- 5d 22h
- Merged PRs (30d)
- 51
Description
This issue has been migrated from [#13456](https://github.com/matrix-org/synapse/issues/13456).
---
Currently we have a number of tables in the database that exist only to record data for the replication streams. These include:
* `cache_invalidation_stream_by_instance`
* `current_state_delta_stream`
* `ex_outlier_stream`
* `presence_stream`
* `push_rules_stream`
(and there may well be others).
Essentially, whenever we record a *change* to a table that needs to be replicated to workers, we add a row to one of these tables; the rows are then used in one of two ways:
* `ReplicationStreamer._run_notifier_loop` regularly polls them (via the `*Stream._update_function` methods) and sends out a `NOTIFY` over [Redis pubsub](https://redis.io/docs/manual/pubsub/) with the data from the table.
* If a worker gets disconnected from Redis (so misses notifications), it can catch up with any missed notification by reading the relevant table itself.
The reason we use this arrangement is twofold:
1. It allows workers which miss the memo (because they were disconnected from Redis) to catch up with anything they missed.
2. Since the Redis notifications are sent out asynchronously by `_run_notifier_loop`, it is possible for the "writing" process to abort between updating the database and sending the notification to Redis. Persisting the data in postgres ensures that we can replay anything that wasn't sent when we restart.
However, I assert that these extra tables are a source of complexity, as well as increased database I/O and storage (not least because we never clear them out (#5888)). Worse, whenever we need to add a new type of replication stream, we have to add a load of extra paraphenalia in the shape of a new stream table. It would be good to consider how to get rid of them.
Contributor guide
Assessment
This issue has not been assessed yet.