element-hq / element-hq/synapse
Ability to deep-merge configuration values (especially for workers)
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 600
- Avg merge
- 5d 22h
- Merged PRs (30d)
- 51
Description
This issue has been migrated from [#11203](https://github.com/matrix-org/synapse/issues/11203).
---
When running a large production deployment of Synapse with many worker process - each with their own config file - one quickly runs into issues with scalability of configuration. Specifically with any non-top level configuration options that need to different between worker processes.
One such example is database arguments. A common configuration is to have different `cp_min` and `cp_max` (database connection minimum and maximum values) for each worker. These are configured with the following config structure:
https://github.com/matrix-org/synapse/blob/1bfd141205d8e2abceef3c277e47f20799bbd455/docs/sample_config.yaml#L773-L783
In order to have a different `cp_*` value for each worker, this entire structure must be copy/pasted between each worker's individual config file. This can be a pain as other, more long-lived options, such as the database host IP (`database.host`), must also be copied over. Then, if one wanted to change the database host IP, they would need to do so across _all_ worker files.
Plainly, if one were to have a full `database` option in the main config, but only:
```yaml
database:
args:
cp_min: 20
cp_max: 30
```
the problem is that the entire `database` section from the main config would be overwritten so that all values other than `database.args.cp_*` would be set to their default. So you end up needing to copy values over.
This issue makes a case for a method of telling Synapse (presumably through a CLI flag) to **deep-merge** config options when multiple config files are present, instead of the current behaviour. This would result in the subset of options shown above preserving those options that were not explicitly overridden.
Deep-merging should be optional and disabled by default, so that any existing config files written for the current behaviour would not be broken.
Contributor guide
Assessment
This issue has not been assessed yet.