element-hq / element-hq/synapse
Long running schema deltas are killed by systemd and lead to bootlooping
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 600
- Avg merge
- 5d 22h
- Merged PRs (30d)
- 51
Description
This issue has been migrated from [#14401](https://github.com/matrix-org/synapse/issues/14401).
---
When a schema delta takes too long, Synapse will get restarted by systemd, possibly repeatedly.
~~When using postgres, schema deltas are transactional. Synapse will run the schema delta and time out again. ~~
~~When using sqlite, schema deltas are not fully transactional (see docstring [here](https://github.com/matrix-org/synapse/blob/8c94dd3a277d4e11192f98a9ca32cb6638606b66/synapse/storage/engines/sqlite.py#L127-L144)). The schema delta will end up being half applied and the database will end up in an inconsistent state. Depending on the nature of the schema delta, the next run may fail.~~ Tracked in #14909.
This manifested as:
* #13193
* #14100
* #14333
Things we may consider doing:
* Write some docs?
* Increase the timeout?
* Pretend to be alive while running schema deltas?
* ~~dmr: Make sqlite schema deltas transactional too?~~ Tracked in #14909.
Contributor guide
Assessment
This issue has not been assessed yet.