element-hq / element-hq/synapse

Long running schema deltas are killed by systemd and lead to bootlooping

Open
#14,401 0 comments 0 reactions 0 assignees View on GitHub
A-Database O-Occasional S-Major T-Defect
Dominant language
Python
Stars
4.6k
Forks
600
Avg merge
5d 22h
Merged PRs (30d)
51

Description

This issue has been migrated from [#14401](https://github.com/matrix-org/synapse/issues/14401).

---

When a schema delta takes too long, Synapse will get restarted by systemd, possibly repeatedly.

~~When using postgres, schema deltas are transactional. Synapse will run the schema delta and time out again. ~~
~~When using sqlite, schema deltas are not fully transactional (see docstring [here](https://github.com/matrix-org/synapse/blob/8c94dd3a277d4e11192f98a9ca32cb6638606b66/synapse/storage/engines/sqlite.py#L127-L144)). The schema delta will end up being half applied and the database will end up in an inconsistent state. Depending on the nature of the schema delta, the next run may fail.~~ Tracked in #14909.

This manifested as:
* #13193
* #14100
* #14333

Things we may consider doing:
* Write some docs?
* Increase the timeout?
* Pretend to be alive while running schema deltas?
* ~~dmr: Make sqlite schema deltas transactional too?~~ Tracked in #14909.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.