element-hq / element-hq/synapse
Docker and scalability
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 600
- Avg merge
- 5d 22h
- Merged PRs (30d)
- 51
Description
This issue has been migrated from [#13555](https://github.com/matrix-org/synapse/issues/13555).
---
**Description:**
I'm currently trying to utilize scaled workers to distribute the workload, mainly for federation, to multiple processes. It turned out, this is more complicated than it needs to be.
See this example docker-compose file:
```yaml
synapse-worker-federation-sender:
<<: *synapse
command: >
run
-m synapse.app.federation_sender
--config-path=/data/homeserver.yaml
--config-path=/data/federation_sender.yaml
labels: [... stuff for reverse proxy ...]
deploy:
mode: replicated
replicas: 4
```
As soon as I start running more than one instance, several issues arise:
- a `worker_name` needs to be specified for each worker
- this needs to be done in the workers config file
- the shared configuration needs to be updated as well
- the main process needs to be restarted as well
Usually it's enough to have one side knowing the details for connecting to its peers. It's either the main process knowing about all worker instances, or all worker instance knowing who's the central controlling unit (preferred). Here we have both sides knowing about both ends, redundant information.
Suggestions:
- Make things like `worker_name` configurable via env vars - to avoid having different, but similar configuration files for the same purpose
- Allow using dynamically generated worker names if nothing is specified - the hostname would be a great start in case of docker containers
- Make workers register themselfes via redis pubsub at runtime.
I'll have to scale up my synapse configuration by a lot in the foreseeable future and I'd really prefer to not define every single worker instance manually. Having at least one of the above mentioned options would make it easier.
I'd also be very happy if I've simply overseen the obvious!
Contributor guide
Assessment
This issue has not been assessed yet.