element-hq / element-hq/synapse
Split different `RDATA` streams into different redis channels
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 600
- Avg merge
- 5d 22h
- Merged PRs (30d)
- 51
Description
This issue has been migrated from [#12461](https://github.com/matrix-org/synapse/issues/12461).
---
Similar to #12460, we should have a separate pub/sub channel for each `RDATA` stream so that workers only need to subscribe to the channels they *actually* need.
This is a bit tricky to do safely, i.e. that workers *do* correctly subscribe to streams they need, especially for caching purposes. I think the best approach is instead of having the replication client call into the various handlers and storage classes:
https://github.com/matrix-org/synapse/blob/e3a49f4784d5c915355ac9306e60b09433db60b5/synapse/replication/tcp/client.py#L134-L148
instead each class registers with the replication client in their `__init__` each stream they care about with a callback to handle that particular streams rows. That way a worker only subscribes to a stream if a class that requires is actually instantiated.
The downside of this is that it's a bit magic, and pulling in a class can have the side effect of causing a stream to suddenly be subscribed to.
Contributor guide
Assessment
This issue has not been assessed yet.