element-hq / element-hq/synapse
[sqlite] Upgrade to 1.68 leads to bootlooping Synapse systemd service
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 600
- Avg merge
- 5d 22h
- Merged PRs (30d)
- 51
Description
This issue has been migrated from [#14100](https://github.com/matrix-org/synapse/issues/14100).
---
### Description
After upgrading Debian packages for `matrix-synapse-py3` (from 1.61.1 to 1.68.0 in this case), Synapse fails to run.
Upon further inspection, it appears that on startup Synapse tries to run a database schema update but doesn't finish and then restarts or gets restarted:
```
synapse.storage.prepare_database - 527 - INFO - main - Applying engine-specific schema 71/01rebuild_event_edges.sql.sqlite
```
Synapse gets killed before being able to finish the db migration.
### Steps to reproduce
- `apt upgrade`
restarting the matrix-synapse service makes no difference and leads to the same result
### Solution
The fix is very simple, tanks to @reivilibre for suggesting it: systemd kills the matrix-synapse service unit for taking too long to report it has started ok (a combination of `Type=notify` and long execution time for that task).
Simply `systemctl edit matrix-synapse.service` and add the following override to increase the timeout and give enough time for the database upgrade to finish:
```
[Service]
TimeoutStartSec=300
```
### Homeserver
personal homeserver
### Synapse Version
1.68.0
### Installation Method
Debian packages from packages.matrix.org
### Platform
Debian 11.5
### Relevant log output
```shell
405072-2022-10-06 14:38:09,765 - synapse.storage.prepare_database - 119 - INFO - main - ['main', 'state']: Existing schema is 71 (+0 deltas)
405073-2022-10-06 14:38:09,765 - synapse.storage.databases.main - 304 - INFO - main - Checking database for consistency with configuration...
405074-2022-10-06 14:38:09,767 - synapse.storage.prepare_database - 411 - INFO - main - Applying schema deltas for v71
405075:2022-10-06 14:38:09,767 - synapse.storage.prepare_database - 527 - INFO - main - Applying engine-specific schema 71/01rebuild_event_edges.sql.sqlite
405076-2022-10-06 17:28:44,945 - root - 343 - WARNING - main - ***** STARTING SERVER *****
```
### Anything else that would be useful to know?
See Solution above
Contributor guide
Assessment
This issue has not been assessed yet.