element-hq / element-hq/synapse
synapse_port_db: IncorrectDatabaseSetup after port was interrupted
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 600
- Avg merge
- 5d 22h
- Merged PRs (30d)
- 51
Description
### Description
I've been getting an error message after synapse_port_db is interrupted (e.g. due to network issues) during the first run.
I'd expect that it is possible to resume by starting port_db again.
### Steps to reproduce
Run synapse_port_db:
`synapse_port_db --sqlite-database /var/lib/matrix-synapse/homeserver.db.snap --postgres-config homeserver-postgres.yaml`
After something broke (e.g. connection issue), reconnect again.
Postgres is accessed over the network.
### Homeserver
selfhosted
### Synapse Version
matrix-synapse-py3/unknown,unknown,now 1.113.0+bullseye1 amd64
### Installation Method
Debian packages from packages.matrix.org
### Database
Both/Migration state
### Workers
Single process
### Platform
Debian 11
### Configuration
database:
name: psycopg2
args:
(removed user/password/host)
# seconds of inactivity after which TCP should send a keepalive message to the server
keepalives_idle: 10
# the number of seconds after which a TCP keepalive message that is not
# acknowledged by the server should be retransmitted
keepalives_interval: 10
# the number of TCP keepalives that can be lost before the client's connection
# to the server is considered dead
keepalives_count: 3
### Relevant log output
```shell
Preparing psycopg2...
2024-08-21 10:23:54,009 - synapse.storage.prepare_database - 135 - INFO - ('main', 'state'): Checking existing schema version
2024-08-21 10:23:54,024 - synapse.storage.prepare_database - 139 - INFO - ('main', 'state'): Existing schema is 86 (+1 deltas)
2024-08-21 10:23:54,024 - synapse.storage.databases.main - 391 - INFO - Checking database for consistency with configuration...
2024-08-21 10:23:54,027 - synapse.storage.prepare_database - 433 - INFO - Applying schema deltas for v86
2024-08-21 10:23:54,034 - synapse.storage.prepare_database - 568 - INFO - Schema now up to date
2024-08-21 10:23:54,040 - synapse.metrics - 121 - WARNING - synapse_background_update_status already registered, reregistering
2024-08-21 10:23:54,380 - synapse.storage.util.id_generators - 98 - INFO - Initialising stream generator for push_rules(id): 25
2024-08-21 10:23:54,383 - synapse.storage.util.id_generators - 98 - INFO - Initialising stream generator for push_rules_enable(id): 28
2024-08-21 10:23:54,445 - synapse_port_db - 926 - ERROR -
Traceback (most recent call last):
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/_scripts/synapse_port_db.py", line 726, in run
self.postgres_store = self.build_db_store(
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/_scripts/synapse_port_db.py", line 642, in build_db_store
store = Store(DatabasePool(hs, db_config, engine), db_conn, hs) # type: ignore[arg-type]
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/event_push_actions.py", line 1896, in __init__
super().__init__(database, db_conn, hs)
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/event_push_actions.py", line 263, in __init__
super().__init__(database, db_conn, hs)
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/client_ips.py", line 84, in __init__
super().__init__(database, db_conn, hs)
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/deviceinbox.py", line 975, in __init__
super().__init__(database, db_conn, hs)
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/devices.py", line 1613, in __init__
super().__init__(database, db_conn, hs)
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/events_bg_updates.py", line 107, in __init__
super().__init__(database, db_conn, hs)
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/media_repository.py", line 114, in __init__
super().__init__(database, db_conn, hs)
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/registration.py", line 2104, in __init__
super().__init__(database, db_conn, hs)
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/registration.py", line 183, in __init__
super().__init__(database, db_conn, hs)
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/room.py", line 1772, in __init__
super().__init__(database, db_conn, hs)
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/room.py", line 152, in __init__
self._un_partial_stated_rooms_stream_id_gen = MultiWriterIdGenerator(
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/util/id_generators.py", line 295, in __init__
self._sequence_gen.check_consistency(
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/util/sequence.py", line 192, in check_consistency
raise IncorrectDatabaseSetup(
synapse.storage.engines._base.IncorrectDatabaseSetup:
Postgres sequence 'un_partial_stated_room_stream_sequence' is inconsistent with associated stream position
of 'un_partial_stated_room_stream' in the 'stream_positions' table.
This is likely a programming error and should be reported at
https://github.com/matrix-org/synapse.
A temporary workaround to fix this error is to shut down Synapse (including
any and all workers) and run the following SQL:
DELETE FROM stream_positions WHERE stream_name = 'un_partial_stated_room_stream';
This will need to be done every time the server is restarted.
Traceback (most recent call last):
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/_scripts/synapse_port_db.py", line 726, in run
self.postgres_store = self.build_db_store(
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/_scripts/synapse_port_db.py", line 642, in build_db_store
store = Store(DatabasePool(hs, db_config, engine), db_conn, hs) # type: ignore[arg-type]
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/event_push_actions.py", line 1896, in __init__
super().__init__(database, db_conn, hs)
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/event_push_actions.py", line 263, in __init__
super().__init__(database, db_conn, hs)
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/client_ips.py", line 84, in __init__
super().__init__(database, db_conn, hs)
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/deviceinbox.py", line 975, in __init__
super().__init__(database, db_conn, hs)
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/devices.py", line 1613, in __init__
super().__init__(database, db_conn, hs)
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/events_bg_updates.py", line 107, in __init__
super().__init__(database, db_conn, hs)
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/media_repository.py", line 114, in __init__
super().__init__(database, db_conn, hs)
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/registration.py", line 2104, in __init__
super().__init__(database, db_conn, hs)
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/registration.py", line 183, in __init__
super().__init__(database, db_conn, hs)
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/room.py", line 1772, in __init__
super().__init__(database, db_conn, hs)
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/room.py", line 152, in __init__
self._un_partial_stated_rooms_stream_id_gen = MultiWriterIdGenerator(
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/util/id_generators.py", line 295, in __init__
self._sequence_gen.check_consistency(
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/util/sequence.py", line 192, in check_consistency
raise IncorrectDatabaseSetup(
synapse.storage.engines._base.IncorrectDatabaseSetup:
Postgres sequence 'un_partial_stated_room_stream_sequence' is inconsistent with associated stream position
of 'un_partial_stated_room_stream' in the 'stream_positions' table.
This is likely a programming error and should be reported at
https://github.com/matrix-org/synapse.
A temporary workaround to fix this error is to shut down Synapse (including
any and all workers) and run the following SQL:
DELETE FROM stream_positions WHERE stream_name = 'un_partial_stated_room_stream';
This will need to be done every time the server is restarted.
Postgres sequence 'un_partial_stated_room_stream_sequence' is inconsistent with associated stream position
of 'un_partial_stated_room_stream' in the 'stream_positions' table.
This is likely a programming error and should be reported at
https://github.com/matrix-org/synapse.
A temporary workaround to fix this error is to shut down Synapse (including
any and all workers) and run the following SQL:
DELETE FROM stream_positions WHERE stream_name = 'un_partial_stated_room_stream';
This will need to be done every time the server is restarted.
```
### Anything else that would be useful to know?
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.