element-hq / element-hq/synapse

[sqlite] Upgrade to 1.68 leads to bootlooping Synapse systemd service

Open
#14,100 0 comments 0 reactions 0 assignees View on GitHub
A-Database O-Occasional S-Minor T-Defect
Dominant language
Python
Stars
4.6k
Forks
600
Avg merge
5d 22h
Merged PRs (30d)
51

Description

This issue has been migrated from [#14100](https://github.com/matrix-org/synapse/issues/14100).

---

### Description

After upgrading Debian packages for `matrix-synapse-py3` (from 1.61.1 to 1.68.0 in this case), Synapse fails to run.

Upon further inspection, it appears that on startup Synapse tries to run a database schema update but doesn't finish and then restarts or gets restarted:

```
synapse.storage.prepare_database - 527 - INFO - main - Applying engine-specific schema 71/01rebuild_event_edges.sql.sqlite
```

Synapse gets killed before being able to finish the db migration.

### Steps to reproduce

- `apt upgrade`

restarting the matrix-synapse service makes no difference and leads to the same result

### Solution

The fix is very simple, tanks to @reivilibre for suggesting it: systemd kills the matrix-synapse service unit for taking too long to report it has started ok (a combination of `Type=notify` and long execution time for that task).

Simply `systemctl edit matrix-synapse.service` and add the following override to increase the timeout and give enough time for the database upgrade to finish:
```
[Service]
TimeoutStartSec=300
```




### Homeserver

personal homeserver

### Synapse Version

1.68.0

### Installation Method

Debian packages from packages.matrix.org

### Platform

Debian 11.5

### Relevant log output

```shell
405072-2022-10-06 14:38:09,765 - synapse.storage.prepare_database - 119 - INFO - main - ['main', 'state']: Existing schema is 71 (+0 deltas)
405073-2022-10-06 14:38:09,765 - synapse.storage.databases.main - 304 - INFO - main - Checking database for consistency with configuration...
405074-2022-10-06 14:38:09,767 - synapse.storage.prepare_database - 411 - INFO - main - Applying schema deltas for v71
405075:2022-10-06 14:38:09,767 - synapse.storage.prepare_database - 527 - INFO - main - Applying engine-specific schema 71/01rebuild_event_edges.sql.sqlite
405076-2022-10-06 17:28:44,945 - root - 343 - WARNING - main - ***** STARTING SERVER *****
```

### Anything else that would be useful to know?

See Solution above

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.