element-hq / element-hq/synapse

Split different `RDATA` streams into different redis channels

Open
#12,461 0 comments 0 reactions 0 assignees View on GitHub
A-Workers T-Task
Dominant language
Python
Stars
4.6k
Forks
600
Avg merge
5d 22h
Merged PRs (30d)
51

Description

This issue has been migrated from [#12461](https://github.com/matrix-org/synapse/issues/12461).

---

Similar to #12460, we should have a separate pub/sub channel for each `RDATA` stream so that workers only need to subscribe to the channels they *actually* need.

This is a bit tricky to do safely, i.e. that workers *do* correctly subscribe to streams they need, especially for caching purposes. I think the best approach is instead of having the replication client call into the various handlers and storage classes:

https://github.com/matrix-org/synapse/blob/e3a49f4784d5c915355ac9306e60b09433db60b5/synapse/replication/tcp/client.py#L134-L148

instead each class registers with the replication client in their `__init__` each stream they care about with a callback to handle that particular streams rows. That way a worker only subscribes to a stream if a class that requires is actually instantiated.

The downside of this is that it's a bit magic, and pulling in a class can have the side effect of causing a stream to suddenly be subscribed to.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.